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Executive  Summary 


This  document  is  an  annotated  version  of  a  seminar  produced  by  IDA  for  the  Office 
of  the  Deputy  Assistant  Secretary  of  Defense  for  Developmental  Test  and  Evaluation. 
The  intent  is  to  provide  general  background  on  electro-optical/infrared  (EO/IR) 
phenomenology  and  systems.  The  paper  begins  with  a  discussion  of  the  factors  affecting 
emission,  transmission,  reflection,  and  absorption  of  light,  including  most  importantly, 
the  role  of  the  atmosphere.  It  continues  with  a  discussion  of  various  applications, 
including  detection  of  point  targets  and  imaging  of  extended  targets.  The  use  of  EO/IR 
for  tracking  targets  is  also  covered,  as  are  some  recent  technological  advancements. 
Finally,  sections  are  dedicated  to  the  topics  of  sampling,  aliasing,  and  reconstruction,  and 
issues  in  color  cameras. 

Because  the  objective  of  the  seminar  was  to  inform  people  with  various  levels  of 
experience  and  range  of  backgrounds,  the  introductory  material  strives  to  provide  the 
basic  building  blocks  for  understanding  the  more  sophisticated  concepts  raised  later. 
This  exposure  should  also  help  Action  Officers  in  daily  activities,  such  as  reviewing  and 
commenting  on  Test  and  Evaluation  Master  Plans  (TEMPs),  test  procedures,  and  test 
result  interpretations. 

Included  in  this  information  are  factors  that  should  be  considered  in  designing  test 
procedures  and  recorded  during  testing  because  they  provide  context  to  the  observed 
results.  These  include  target-background  contrast  and  intensity  of  the  illumination  source 
(for  bands  where  reflected  light  is  dominant),  target  temperature  (for  bands  where  emitted 
radiation  dominates),  visual  range,  and  operating  altitude  of  the  platform  on  which  the 
EO/IR  sensor  is  integrated. 

Performance  of  an  EO/IR  sensor  depends  upon  the  optics,  detector,  and  display,  in 
addition  to  factors  mentioned  above.  Hence  it  is  unwise  to  evaluate  the  potential  utility 
of  an  EO/IR  sensor  from  specifications  alone,  that  is,  without  using  a  detailed 
engineering  model.  Nevertheless,  all  other  things  being  equal,  it  can  be  argued  that,  for 
an  imaging  sensor  designed  to  recognize  or  identify  targets,  it  is  better  to  have  a  focal 
plane  array  with  smaller  detector  elements,  assuming  the  optics  modulation  transfer 
function  (MTF)  is  not  limiting  the  overall  system  MTF.  This  follows  because  the 
improved  resolution  of  such  a  design  will  enhance  range  performance  if  the  ground 
sample  distance  is  the  limiting  factor.  In  a  similar  “rule  of  thumb”  vein,  optics  with 
larger  focal  lengths  provide  the  potential  for  better  resolution,  assuming  the  detector’s 
MTF  is  not  limiting  the  overall  system  MTF.  This  comes  at  the  expense  of  reducing  the 
sensor’s  overall  field-of-view.  However,  we  stress  that  it  is  difficult  to  anticipate  a  priori 
how  all  of  the  factors  affecting  image  quality  interact;  hence  we  recommend  the  use  of 
modeling  and  detailed  system  analysis  to  interpret  potential  sensor  performance. 


This  document  is  divided  into  seven  sections  as  follow: 


1 .  EO/IR  Basics 

2.  Non- Imaging  EO/IR  Systems 
(Point  Targets) 

3.  Imaging  EO/IR  Systems 
(Extended  Targets) 


4.  Tracking 

5.  Latest  Trends  in  EO/IR  System 
Development 

6.  Aliasing 

7.  Issues  in  Bayer  Color  Cameras 


A  brief  summary  of  each  section  follows. 


EO/IR  Basics 

EO/IR  systems  cover  a  wide  range  of  distinct  technologies  based  on  the  targets  and 
mission  to  be  accomplished.  Target  phenomenology  often  dominates  the  choice  of 
spectral  band.  For  example,  missile  launch  detection  depends  on  the  very  hot  missile 
exhaust,  which  produces  significant  radiation  in  the  ultraviolet  (UV)  spectral  region.  The 
band  choice  is  also  influenced  by  the  vagaries  of  atmospheric  transmission  and  scattering. 
EO/IR  missions  divide  roughly  into  dealing  with  point  targets  and  extended  fully  imaged 
targets.  For  point  targets,  the  challenge  is  to  extract  the  target  from  a  complex  and 
cluttered  background.  For  fully  imaged  targets  (images  of  tanks,  for  example),  the 
contrast  between  target  and  background  is  a  critical  parameter.  The  ability  to  detect  dim 
targets,  either  point  or  extended,  is  a  measure  of  the  system’s  sensitivity.  The  resolution 
of  a  sensor  is  a  measure  of  its  ability  to  determine  fine  detail.  Measures  of  resolution 
depend  on  the  precise  task.  Use  of  eye-chart-like  calibration  targets  is  common  in  DoD 
applications.  EO/IR  sensors  may  be  divided  into  scanning  sensors,  which  use  a  limited 
number  of  detectors  to  scan  across  the  scene,  and  staring  sensors,  which  use  large 
numbers  of  detectors  in  rectangular  arrays. 


Non-Imaging  EO/IR  Systems 

Non-imaging  point  target  EO/IR  systems  focus  on  the  task  of  detecting  targets  at 
long  range.  For  these  applications,  details  of  the  target  are  irrelevant;  for  example,  for  IR 
sensors,  only  the  total  energy  emitted  matters,  not  the  precise  temperature  and  size  of  the 
target  separately.  The  available  energy  can  vary  over  many  orders  of  magnitude,  from 
the  engine  of  a  small  UAV  to  an  intercontinental  ballistic  missile  (ICBM)  launch. 
Comparison  of  the  available  energy  at  the  sensor  to  the  noise  level  of  the  sensor  provides 
the  central  metric  of  sensor  performance,  the  noise  equivalent  irradiance  or  NEI.  The 
problem  of  extracting  the  target  from  background  clutter  is  addressed  in  two  basic  ways: 
through  the  choice  of  sensor  band  and  algorithms.  For  missile  launch  detection,  for 
example,  one  may  use  the  “solar  blind”  UV  band.  The  transmission  in  this  band  is  very 
poor,  and  the  general  background  is  reduced  to  a  level  of  near  zero.  However,  the  very 
bright  missile  signature  is  strong  enough  to  still  provide  a  useable  signal.  Algorithms  for 
separating  a  point  target  from  a  structured  background  are  a  continuing  source  of 
improvement  in  performance,  but  their  complexity  may  make  their  evaluation 
challenging. 
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The  effectiveness  of  imaging  systems  can  be  degraded  by  many  factors,  including 
limited  contrast  and  luminance,  the  presence  of  noise,  and  blurring  due  to  fundamental 
physical  effects.  As  a  mathematical  description  of  image  blur,  the  MTF  can  be  broken 
down  into  each  component  of  the  sensing,  such  as  optics,  detector,  atmosphere,  and 
display.  This  provides  insight  into  the  sources  and  magnitude  of  image  degradation. 

There  are  many  ways  to  evaluate  and  describe  the  quality  of  an  image,  the  use  of 
which  is  determined  by  details  of  the  application.  Two  examples  have  a  broad 
application  and  a  long  history. 

1.  The  Army  Night  Vision  and  Electronic  Sensors  Directorate  (NVESD)  maintains 
models  for  predicting  image  quality  based  on  the  targeting  task  performance 
(TTP)  metric. 1  This  provides  a  rule-of-thumb  for  the  number  of  bar  pairs 
required  on  target  to  have  a  50-percent  probability  of  detecting,  recognizing,  or 
identifying  the  target. 

2.  The  strategic  intelligence  community  uses  the  National  Imagery  Interpretability 
Rating  Scale  (NIIRS)  to  evaluate  the  quality  of  still  imagery.  NIIRS  is  a 
qualitative  scale,  running  from  0  to  9.  Tables  are  published  associating  rating 
levels  with  tasks  for  EO,  IR,  synthetic  aperture  radar  (SAR),  civil  imagery, 
multispectral  imagery,  and  video.  NIIRS  can  be  estimated  by  the  General  Image 
Quality  Equation  (GIQE),  an  empirical  model  that  has  been  designed  to  predict 
NIIRS  from  sensor  system  characteristics. 

Each  of  these  approaches  represents  a  balance  between  two  of  the  fundamental 
limits  to  any  EO  sensor:  the  resolution  (how  small)  and  the  sensitivity  (how  dim)  -  that 
is,  how  small  an  object  or  feature  can  be  usefully  seen  and  how  low  can  the  signal  be 
before  it  is  overwhelmed  by  the  noise.  Figure  ES-1  shows  a  comparison  of  a  noise- 
limited  case  and  a  resolution-limited  case. 


ES-1.  Noise  and  Resolution  Limited  Images 


1  The  TTP  metric  is  the  successor  of  the  Johnson  criteria.  See  page  30  for  additional  discussion. 
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In  modem  digital  sensors,  these  classical  factors  have  been  joined  by  a  third,  the 
sampling  or  pixelation  of  the  sensor,  as  shown  in  Figure  ES-2. 


ES-2.  Sampling  Limited  Image 


The  design  of  a  sensor  is  a  tradeoff  between  these  three  factors,  placed  in  the 
context  of  characteristics  of  the  target  and  environment.  While  in  a  balanced  design,  each 
of  the  factors  needs  to  be  evaluated.  For  any  given  sensor  and  state  of  the  technology 
employed,  it  may  be  the  case  that  the  sensor  is  signal-to-noise  limited;  in  this  case,  the 
S/N  ratio  is  the  single-most  important  parameter  and  rules-of-thumb  may  be  developed  to 
focus  on  S/N  for  the  target  and  environment.  In  other  cases,  the  sensor  is  resolution 
limited,  and  the  characterization  of  the  optics  (the  MTF)  is  the  most  important;  an  ideal 
sensor  is  said  to  be  diffraction-limited,  able  to  respond  to  spatial  frequencies  up  to  the 
diffraction  limit,  A/D,  where  A  is  the  wavelength  at  which  the  sensor  operates  and  D  is  the 
diameter  of  the  optics.  For  these  cases,  rules-of-thumb  may  be  developed  that  focus  on 
the  optical  system.  For  sensors  that  are  limited  by  the  pixelation,  the  “Nyquist  limit”  is 
determined  by  the  separation  between  detectors,  S;  the  highest  spatial  frequency  that  can 
be  correctly  measured  by  the  sensor  is  1/2S.  In  these  cases  the  Nyquist  frequency  may  be 
an  appropriate  rule-of-thumb. 

These  trades  can  be  followed  in  the  history  of  personal  cameras  from  the  film  era  to 
the  most  modem  Nikon  or  Canon.  In  the  film  era,  the  user  could  trade  between 
sensitivity  and  resolution  by  changing  the  film  used:  so-called  fast  film  (TRI-X)  was 
sensitive  but  had  a  coarse  grain  size,  limiting  the  resolution.  It  was  useful  for  pictures  in 
dim  light  or  when  high  shutter  speeds  to  photograph  moving  subjects  were  needed.  In 
brighter  light  or  for  slowly  moving  objects  the  photographer  might  choose  a  slow  film 
(PAN-X)  that  had  a  fine  grain  size  but  was  not  very  sensitive.  On  top  of  this  choice,  the 
photographer  could  choose  lenses  with  different  f-numbers  (to  change  sensitivity), 
different  focal  lengths  (to  change  resolution),  and  different  prices  (to  improve  MTF). 

In  the  last  2  decades,  digital  cameras  have  replaced  film  cameras  in  many 
applications.  The  earliest  digital  cameras  were  limited  both  by  sensitivity  (detectors  were 
noisy)  and  by  sampling.  In  consumer  terms,  the  sampling  resolution  was  specified  by  the 
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number  of  pixels;  early  cameras  were  1  megapixel,  about  1,000  by  1,000  detectors 
replacing  the  film.  The  need  for  relatively  large  detectors  (10-20  pm)  was  driven  by  lack 
of  detector  sensitivity;  the  resulting  reduction  in  resolution  compared  to  the  better  films 
that  had  film  grains  of  2-3  pm  was  a  consequence  of  this,  rather  than  a  technological  limit 
to  producing  smaller  detectors.  As  detector  technology  evolved,  the  noise  in  each 
detector  was  reduced  (improving  S/N  and  allowing  dimmer  light  or  faster  motion)  and 
the  number  of  pixels  was  increased.  Today,  personal  cameras  can  have  36  megapixels  or 
more  and  begin  to  rival  the  finest  grain  films  for  resolution  with  detectors  of  4  pm  or 
smaller.  The  best  personal  cameras  began  as  sensitivity  and  sampling  limited;  but  as 
those  barriers  were  overcome,  they  have  returned  to  being  resolution  limited  as 
determined  by  the  quality  of  the  optics.  As  these  changes  have  taken  place,  the  simple 
rules-of-thumb  that  characterize  the  camera  have  also  changed. 

The  same  three  elements  are  present  in  all  EO  systems,  but  there  is  no  unique 
balance  point  between  them.  This  is  determined  by  the  state  of  technology  but  also  the 
mission  the  sensor  is  trying  to  accomplish.  Personal  cameras  emphasize  images  of 
people  at  close  to  moderate  ranges;  the  signal  is  determined  by  reflected  light,  the  noise 
by  the  technology  of  the  detectors.  Missile  launch  detection  systems  are  looking  for 
extremely  bright  points  of  light  in  a  cluttered  background.  Infrared  systems  depend  on 
the  self-emission  of  the  target  (hot  engines  on  aircraft  and  tanks)  and  are  often  limited  by 
the  environment  (the  transmission  of  the  signal  through  the  air).  No  matter  what  the 
sensor  is,  the  evaluation  of  the  sensor  system  involves  characterizing  it  with  the  three 
metrics  of  noise,  resolution,  and  sampling. 

Tracking 

EO/IR  sensor  systems  are  often  paired  with  a  tracker.  Examples  of  point  target 
trackers  include  systems  to  track  missile  launches  and  infrared  search  and  track  systems, 
both  discussed  above.  Many  different  algorithms  are  used  to  implement  point  trackers. 
Although  different,  they  all  involve  the  same  basic  steps:  track  initiation,  data 
association,  track  smoothing,  and  track  maintenance.  Conversely,  image-based  trackers 
are  used  to  track  mobile  objects  on  the  ground  [e.g.,  intelligence,  surveillance,  and 
reconnaissance  (ISR)  missions]  or  for  aim  point  selection.  Usually  with  image-based 
trackers,  the  target  is  initially  located  by  a  human  operator  viewing  a  display.  After  the 
target  is  acquired,  a  feedback  control  loop  (track  loop)  continuously  adjusts  the  gimbal  to 
keep  the  target  in  the  center  of  the  sensor’s  field  of  view.  Common  image-based  tracker 
algorithms  include  edge  detection,  centroid  tracking,  area  correlation  tracking,  moving 
target  indicators  (MTI),  multi-mode  trackers,  and  feature-based  algorithms. 

Latest  Trends  in  EO/IR  System  Development 

Driven  by  missions  such  as  Persistent  Surveillance,  an  important  recent  trend  in 
EO/IR  sensors  is  the  development  of  large-area,  high-resolution  cameras.  For  sensors 
operating  in  the  visible  band,  this  is  simpler  than  for  those  operating  in  the  mid- 
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wavelength  infrared  (MWIR)  or  long-wavelength  infrared  (LWIR)  bands.  In  the  visible 
band,  silicon  can  be  used  for  both  sensing  material  and  for  readout  electronics.  This 
eliminates  issues  such  as  potential  thermal  mismatch  between  the  two.  For  EO  systems, 
it  is  also  possible  to  group  individual  cameras  together  and  then  combine  separate  images 
into  one.  The  current  state  of  the  art  in  MWIR  focal  plane  arrays  (FPAs)  is  roughly  4,000 
by  4,000  detectors;  for  LWIR  FPAs,  it  is  closer  to  2,000  by  2,000.  The  ARGUS  sensor 
being  developed  by  DARPA  has  1.8  gigapixels.  To  reduce  the  amount  of  thermally 
generated  noise  in  LWIR  FPAs,  these  cameras  operate  at  low  temperatures.  The  required 
cryogenic  equipment  has  important  Size,  Weight,  and  Power  (SWaP)  implications  at  the 
system  level  and  can  affect  system  reliability.  To  address  this  need,  considerable  effort 
has  gone  into  development  of  uncooled  sensors.  Another  recent  development  is  the  use 
of  so-called  III-V  superlattices  that  use  alternating  layers  such  as  aluminum  gallium 
arsenide  and  gallium  arsenide  (AlGaAs/GaAs)  and  that  allow  one  to  change  the  device’s 
spectral  response  by  changing  individual  layer  thicknesses.  The  short-wavelength 
infrared  (SWIR)  band  extends  from  1.0  pm  to  3.0  pm.  This  radiation,  often  referred  to  as 
“airglow”  or  “nightglow,”  is  non-thermal  radiation  emitted  by  the  Earth’s  atmosphere. 
Military  applications  of  SWIR  include  low-light/night-vision  imaging,  detection  of  laser 
designators,  covert  laser  illumination,  laser-gated  imaging,  threat-warning  detection,  ISR, 
and  covert  tagging.  Hyperspectral  imaging  allows  one  to  simultaneously  gather  both 
spatial  and  spectroscopic  information  for  more  accurate  segmentation  and  classification 
of  an  image. 

Aliasing 

Aliasing  is  an  almost  inescapable  feature  of  staring  focal  plane  array  sensors. 
Aliasing  refers  to  the  artifacts  associated  with  sampling  systems.  They  can  range  from 
Moire  patterns2  to  ambiguously  interpreted  images  of  objects  with  complicated  patterns 
of  light  and  dark.  The  presence  of  aliasing  in  a  system  gives  rise  to  a  general  rule  based 
on  the  Nyquist  frequency,  which  identifies  the  resolution  of  a  system  with  the  size  of  a 
sample  (or  pixel)  on  the  target.  This  underrates  the  importance  of  other  determiners  of 
resolution  such  as  the  MTF.  Performance  may  be  better  or  worse  than  that  suggested  by 
Nyquist,  depending  on  the  task.  The  combination  of  MTF  and  aliasing  can  lead  to  some 
unintuitive  tradeoffs.  For  example,  poorer  MTF  may  actually  improve  performance  in 
some  tasks  if  aliasing  is  a  significant  problem.  For  imaging  systems  with  aliasing,  the 
display  of  the  sensor  information  becomes  important,  especially  if  the  image  is  zoomed 
so  that  one  display  pixel  is  used  to  represent  a  single  sensor  sample  value.  An  improper 
reconstruction  of  the  sensor  data  can  degrade  performance.  For  all  aliased  systems,  an 
understanding  of  the  reconstruction  algorithm  is  an  important  aspect  of  any  system 
including  a  human  operator. 


2  Moire  patterns  are  interference  patterns  that  are  often  the  result  of  digital  imaging  artifacts.  See  page  59 
for  additional  discussion. 
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Issues  in  Bayer  Color  Cameras 

The  use  of  Bayer  pattern  color  cameras  in  imaging  systems  to  reduce  cost  and  data 
rate  requirements  may  introduce  additional  dependence  on  algorithmic  choices.  In 
contrast  to  full  color  cameras,  a  Bayer  camera  does  not  have  a  color  detector  for  all  3 
colors  (red,  green,  blue)  at  each  location  (a  12  “megapixel”  Bayer  sensor  has  6  million 
green,  3  million  red,  and  3  million  blue  detectors).  An  algorithm  is  used  to  interpolate 
the  missing  values  to  produce  a  complete  three-color  image.  This  color  aliasing  is  more 
severe  than  the  aliasing  intrinsic  to  any  staring  focal  plane.  Fortunately,  in  the  natural 
world,  color  information  is  usually  relatively  immune  to  aliasing  effects.  However,  a 
large  number  of  different  algorithms  of  varying  complexity  may  be  employed  by  a  sensor 
manufacturer.  Depending  on  the  tasks  to  be  performed  and  the  details  of  the  image,  the 
algorithm  may  induce  new  artifacts  that  complicate  system  evaluation.  Ideally,  the  user 
could  be  given  some  control  over  the  interpolation  algorithm  to  match  it  to  the  task. 


(This  page  intentionally  left  blank.) 
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A  Tutorial  on  Electro-Optical/Infrared  (EO/IR) 
Theory  and  Systems 


EO/IR  Missions 

•  Point  targets 

-  IRST  (infrared  search  and  track)  and  visible  analogs.  Detection  and 
tracking  of  aircraft  at  long  range  against  sky  backgrounds. 

•  Ship-based  and  fighter-based  IRST  systems  have  been  developed  in  the 
past. 

•  Applications  to  homeland  security  and  anti-UAV. 

-  Missile  launch  warning.  Detection  of  missile  launch  events  and 
tracking  against  ground  backgrounds. 

•  Satellite  and  aircraft  based 

•  Extended  targets 

-  ISR  systems 

•  Satellite,  manned  aircraft,  unmanned  aircraft 

•  Any  and  all  bands 

•  Usually  oriented  close  to  vertically  or  side-to-side 

-  Target  acquisition  and  surveillance 

•  Night  vision  goggles 

•  Vehicle  or  aircraft  mounted  FLIRs  (Forward  Looking  Infrared) 

•  Usually  oriented  approximately  horizontally  with  variable  look-up  or  look- 
down  angles 


The  most  basic  division  in  EO/IR  sensors  is  between  point  and  extended  targets.  A 
point  target  is  one  that  is  small  enough,  compared  to  the  resolution  capabilities  of  the 
sensor,  to  appear  as  a  single  point.  The  defining  characteristic  of  an  extended  target  is 
the  presence  of  internal  detail.  Rather  than  detecting  a  point,  the  sensor  presents  an 
image  of  the  target.  The  distinction  between  point  and  imaging  targets  is  not  the  same  as 
between  non-imaging  and  imaging  sensors.  One  can  imagine  a  point  target  mission  that 
scans  the  sky  looking  for  a  bright  spot  without  ever  forming  an  image  (the  original 
Sidewinder  air-to-air  missile  seeker  would  be  such  as  system);  however,  some  imaging 
sensors  form  an  image  of  the  sky  and  then  look  for  bright  spots  within  the  image. 

Infrared  search  and  track  (IRST)  systems  represent  a  class  of  systems  for  which  the 
targets  remain  point  targets.  The  term  primarily  refers  to  IR  sensors  dedicated  to 
detecting  aircraft  at  long  range.  Aircraft  are  typically  hotter  than  the  sky  or  cloud 
background  against  which  they  are  seen  and  appear  as  a  bright  point.  Although  originally 
conceived  as  an  air-to-air  aircraft  finder  or  a  ship-to-air  missile  finder  (to  detect  anti-ship 
cruise  missiles),  the  same  approach  has  applications  for  homeland  security  and  anti¬ 
unmanned  aerial  vehicle  (UAV)  search.  Another  class  of  systems  is  missile-launch 
detection  systems.  In  this  case,  the  launch  is  signaled  with  a  very  hot  flash  that  can  be 
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detected.  In  this  case,  the  background  is  likely  to  be  the  ground  rather  than  the  sky  for 
IRST  systems;  this  affects  the  algorithms  used  to  extract  the  target  and  even  the  band 
(wavelength)  of  the  EO/IR  sensor. 

The  extended  target  applications  include  intelligence,  surveillance,  and 
reconnaissance  (ISR)  systems  providing  long-term  imaging  of  the  ground,  usually  from  a 
vertical  perspective  (e.g.,  surveillance  satellites,  UAY),  and  target  acquisition  systems  in 
tactical  systems  [night  vision  goggles,  tank  forward-looking  infrared  (FLIR)  systems], 
typically  a  more  nearly  horizontal  view. 
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Parameters 

*  For  all  EO/IR  systems 

-  Optical  properties  (resolution) 

-  Sensitivity/noise 

-  Digital  artifacts 

-  Detector  properties 

-  Processing 

*  The  importance  of  some  parameters  varies  with  mission 

-  Target  characteristics 

-  Platform  characteristics 

-  For  automated  systems  (IRST),  algorithms  for  detecting  targets  (high 
probability  of  detection  with  limited  false  alarms) 

-  For  observer-in-the-loop  systems,  displays 

*  For  example,  ISR  systems 

-  Very  high  resolution  (small  instantaneous  field  of  view,  IFOV)  to  achieve  good 
ground  resolution  from  high  altitudes 

-  Field  of  view 

•  A  100-megapixel  array  with  1-foot  ground  sample  distance  covers  only  a  2  mile  x  2 
mile  patch  of  ground 

•  Older  film  cameras  have  140  degree  fields  of  view  with  a  72  mile  wide  x  2  mile  image! 


The  definition  of  each  of  the  terms  mentioned  above  will  be  developed  over  the 
course  of  this  paper. 

It  may  be  surprising  to  see  film  discussed  in  the  21st  century;  however,  in  some 
ways,  film  remains  unsurpassed. 

Developed  in  the  early  1960s,  the  U-2’s  Optical  Bar  Camera  (OBC)  gave  the  spy 
plane  an  edge  that  has  yet  to  be  matched  by  its  prospective  successor:  the  Global  Hawk 
unmanned  aircraft.  The  film  camera,  which  is  mounted  in  a  bay  behind  the  aircraft’s 
cockpit,  captures  high-resolution  panoramic  images  covering  an  area  140  degrees  across. 

The  OBC  gets  its  name  from  its  slit  assembly,  which  forms  the  camera’s  bar. 
Unlike  a  conventional  camera,  in  which  an  aperture  opens  and  closes  to  allow  light  to 
expose  the  film  (or,  in  the  case  of  a  digital  camera,  a  charge  coupled  device  chip),  the  slit 
exposes  a  roll  of  film  mounted  on  a  3 -foot-diameter  rotating  spindle.  The  entire  lens 
assembly,  with  a  30-inch  lens,  rotates  around  the  longitudinal  axis  of  the  aircraft  to 
provide  a  field  of  view  that  extends  70  degrees  to  either  side  of  the  aircraft’s  flight  path. 

The  OBC  carries  a  roll  of  film  measuring  some  two  miles  in  length  and  weighs 
around  100  pounds.  Each  frame  is  63  inches  long  by  5  inches  wide. 
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A.  Basics 


The  EO/IR  Sensing  Process 


Pixel  Sampling 
Optical  Blur 
Optical  Size 
Optics  /  Pixel  Losses 
Jitter  /  Stability 


Illumination  of  Target 
Natural  or  Artificial 


Reflection  of  incident  light 
Emission  of  light  due  to  target 
temperature  and  emissivity 


Target  Size 
Target  Contrast 


Illumination 

Atmospherics 

Target 

Background/Clutt 

er 

Optics/Detector 

Datalink 

Processing 

Display/Observer 


Type  of  Display 
Observer  (eye) 
Capabilities 
Contrast  Enhancement 
Other  Image  Processing 
atic  processing 


EO/IR  systems  are  often  plagued  by  the  difficulty  of  providing  precise  predications 
of  performance:  in  EO/IR,  the  answer  is  “It  depends.” 

The  majority  of  EO/IR  systems  (laser  systems  are  the  conspicuous  exception,  but 
searchlights  are  still  used  as  well,  and  night  vision  goggles  can  be  used  with  artificial 
illumination)  are  passive;  that  is,  they  are  dependent  on  the  illumination  of  the  target  by 
sunlight  or  moonlight  (even  sky  shine)  or  the  target’s  own  emission  of  light.  This  is  in 
contrast  to  radar  systems  that  provide  their  own  illumination.  In  addition,  the  EO/IR 
photons  travel  through  a  hostile  medium,  the  atmosphere,  which  can  refract,  absorb,  or 
scatter  them.  EO/IR  is,  in  general,  sensitive  to  the  environment.  Also,  the  target’s 
characteristics  can  be  quite  important  and  variable,  depending  on  the  target  temperature 
and  even  the  paint  used  on  the  target.  This  variability  poses  a  problem  to  the  designer, 
tester,  and  operator. 

The  designer  has  more  control  over  the  sensor’s  design  parameters,  datalinks  used  to 
transfer  information,  and  displays  on  which,  for  imaging  sensors,  the  scenes  are  presented 
to  the  human  operator. 
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Electro-optical/  Infrared  Systems 


EO/IR  systems  cover  the  range  from  UV  through  visible  and  IR. 


0.38  -0.75  m 


•  The  wide  range  of  EO/IR  systems  share  some  similarities: 

-  Lenses  and  mirrors  rather  than  antennas 

-  Dependence  on  the  environment 

•  Although  film  systems  remain  in  the  inventory  and  are  still 
preferred  for  some  missions,  new  EO/IR  systems  will  be  digital. 

•  The  band  chosen  reflects  the  phenomenology  of  the  target. 


Image  Source: 

D.  G.  Dawes  and  D.  Turner, 
"Some  Like  It  Hot," 
Photonics  Spectra, 
December  2008. 


LWIR:  emitted  radiation  SWIR:  reflected  radiation 


The  category  of  EO/IR  sensors  extends  from  the  ultraviolet  at  a  wavelength  of  0.25 
micrometers  ( gim )  (which  can  be  useful  for  very  hot  missile  launch  detection 
applications)  through  the  visible  region  to  the  infrared  region  (wavelengths  up  to  14  pm 
or  even  greater).  The  design  wavelength  band  is  determined  by  the  expected  targets  and 
mission. 
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Reflected  and  Emitted  Light 

•  The  energy  coming  from  the  target  and  background  is  a 
mixture  of  reflected  and  emitted  light. 

-  Reflected  light  is  the  predominate  element  for  visible  sensors  but 
also  contributes  in  near-IR,  short  wave-IR,  and  mid-wave-IR. 

•  Depends  on  natural  or  artificial  illumination  (for  example,  night  vision 
goggles  rely  on  ambient  light  in  the  near-IR). 

-  With  artificial  illumination,  essentially  any  wavelength  may  lead  to 
significant  reflection. 

-  Emitted  light  is  a  natural  feature  of  all  objects:  the  amount  of  light 
radiated  and  at  what  wavelengths  it  is  concentrated  depends  on 
the  temperature  and  emissivity  of  the  object. 

-  Reflectance  and  emissivity  vary  with  wavelength. 

•  Snow  is  highly  reflective  in  the  visible  but  highly  emissive  in  the  IR. 

-  Incident  radiation  is  either  reflected,  transmitted,  or  absorbed. 

•  For  opaque  objects,  the  radiation  is  either  reflected  or  absorbed. 

•  In  equilibrium  energy  that  is  absorbed  is  also  emitted  (but  perhaps  at  a 
different  wavelength). 


As  noted  in  the  chart,  the  choice  of  wavelength  is  connected  to  the  balance  of 
emitted  versus  reflected  light.  One  obvious  difference  is  that  human  operators  are 
accustomed  to  how  objects  appear  in  reflected  visible  light.  The  short  wave  IR,  which 
almost  entirely  depends  on  reflected  light,  produces  images  that  are  very  like  visible 
images;  mid-wave  IR,  less  so;  and  for  long-wave  IR,  the  images  seem  unnatural  in  some 
aspects. 
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Planck’s  Law 

•  Planck’s  law  determines  the  energy  radiated  as  a  function  of  the 
wavelength,  temperature,  and  emissivity  (e).  The  peak 
wavelength  varies  with  temperature:  higher  temperature  => 
shorter  wavelengths. 

-  For  objects  near  room  temperature  (300  K),  the  radiation  peaks  in 
the  LWIR  (long  wave  Infrared:  8-12  jam).  For  most  natural  objects  s  is 
close  to  1 . 

-  In  MWIR  (mid-wave  infrared:  3-5  pm)  and  most  natural  scene 
objects  there  is  a  mixture  of  emitted  and  reflected  light.  Emitted 
component  increases  with  higher  temperature. 

-  In  SWIR  (short-wave  infrared:  1-3  pm),  there  is  a  mixture  of  emitted 
and  reflected  radiation. 

-  NIR  (near  infrared:  0.7-1 .0  pm)  depends  upon  reflected  light  and 
artificial  illumination. 

-  In  the  visible,  the  radiance  is  almost  entirely  reflected  light  for 
typical  temperatures  (but  things  can  be  red  or  white  hot!). 

-  UV  applications  depend  primarily  on  emitted  energy  of  very  hot 
objects. 

•  Radiance  is  conventionally  measured  in  either: 

-  Watts/cm2/ster;  or 

-  Photons/sec  /cm2/ster  L  hc 

^  watts  photons  » 

sensor 


Emitted  radiance  (L)  of  blackbody3  with  temperature  T  and  emissivity  8  between  X\ 
and  X2  is  given  by: 

2  c  1 


^watts 


-/ 


photons 

A"? 


-f 

2h  c2 


^4  ghc/kTA  _  ^ 

1 


d; l 


A5  ehc/kTA  _1dA~  ^photons  ^ 


he 


For  most  modem  EO/IR  sensors,  the  photon  representation  is  the  more  appropriate 
because  the  detectors  are  essentially  photon  counters.  However,  very  early  IR  systems 
and  some  modem  uncooled  systems  are  bolometers,  measuring  the  energy  of  the  signal 
rather  than  the  number  of  photons.  For  these  systems,  the  radiance  in  terms  of  watts  is 
the  more  useful.  The  terminology  can  be  quite  confusing  and  care  must  be  taken  to  keep 
the  units  straight. 


3  A  blackbody  is  an  idealized  physical  body  that  absorbs  all  incident  electromagnetic  radiation,  regardless 
of  frequency  of  angle  of  incidence.  A  blackbody  at  constant  temperature  emits  electromagnetic  radiation 
called  blackbody  radiation.  The  radiation  is  emitted  according  to  Plank’s  law,  meaning  that  it  has  a 
spectrum  that  is  determined  by  the  temperature  alone  and  not  by  the  body’s  shape  or  composition. 

(wiki) 
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Atmospheric  Effects 

•  Only  the  radiance  within  specific  atmospheric  transmission 
windows  is  usually  useful:  in  some  cases,  systems  exploit 
edges  of  the  atmospheric  window. 

•  The  atmosphere  absorbs  energy,  particularly  in  IR  and  UV. 

-  The  details  of  the  absorption  depend  on  the  water  vapor,  C02, 
smokes.... 

-  The  deep  valleys  (for  example  between  4-4.5  jam,  at  9.5  jim) 
define  the  broad  bands  but  there  is  a  lot  of  fine  structure  that 
can  be  important  for  some  applications. 

-  These  determine  the  transmission,  t,  of  the  signal  through  the 
atmosphere. 

•  The  atmosphere  scatters  energy,  particularly  in  the  visible 
(fog,  haze). 

-  Depends  strongly  on  the  distribution  of  aerosol. 

-  Energy  from  the  target  scattered  out  of,  other  energy  scattered 
into,  the  line  of  sight. 

•  The  atmosphere  also  radiates  (primarily  in  the  LWIR):  this  is  called 
“path  radiance.” 

•  Each  of  these  reduce  the  contrast  between  the  target  and  the 
background. 

•  Elaborate  models  have  evolved  to  predict  transmission,  scattering, 
and  path  radiance,  paced  by  the  improvement  in  computational 
speeds. 


The  complicated  structure  of  the  atmospheric  transmission  curves  reflects  the  fact 
that  the  atmosphere  is  a  mixture  of  different  molecules  with  each  species  having  its  own 
set  of  absorption  and  emission  lines.  Because  it  is  a  mixture,  the  curves  will  depend  on, 
for  example,  the  absolute  humidity  (the  water  content  of  the  air).  Pollutants  such  as 
ozone  or  discharges  from  factories  can  be  relevant,  as  can  aerosols  suspended  in  the  air, 
including  water  (ordinary  haze  and  fog),  sand,  dirt,  and  smoke. 

Although  it  is  common  to  assume  a  simple  exponential  decay  of  transmission  with 
range  (so-called  Beer’s  Law),  it  is  never  safe  to  do  so  without  checking  the  more 
elaborate  calculations  involving  individual  absorption  lines,  rather  than  averages  over 
bandwidth  ranges.  The  dependence  of  EO/IR  on  transmission  is  the  primary  factor  that 
makes  the  predicting  and  testing  of  EO/IR  systems  challenging. 
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Contrast  and  Clutter 

•  Separating  targets  from  the  background  depends  on  the  contrast: 

is  the  target  brighter/darker  than  the  background? 

-  The  contrast  is  reduced  by  the  transmission  loss  and  may  be 
masked  by  scattered  energy  or  path  radiance. 

-  Transmission  loss  increases  with  the  range  and  is  frequently  I  | 
limiting  factor  in  sensor  performance  (overwhelming  naive  resolution 
estimates). 

-  Atmospheric  variation  is  a  primary  cause  of  variability  in  performance. 

-  EO/IR  sensors  that  perform  well  in  clear  weather  will  be  degraded  by 
dust,  smoke,  clouds,  fog,  haze,  and  even  high  humidity. 

•  Even  with  sufficient  contrast,  the  target  must  be 

extracted  from  the  background. 

-  Non-imaging  or  point  target  detection:  locating  the 
dot  in  the  complicated  image  (there  are  4  added  dots 
in  image). 

-  Imaging  systems:  the  people  in  the  foreground  are 
easy  to  see,  but  are  there  people  in  the  trees? 

•  These  concepts  will  be  revisited  in  more  detail  in 

later  charts. 


Contrast  can  be  defined  in  different  ways  depending  on  the  application.  At  its  most 
basic,  it  is  simply  the  difference  in  sensor  output  between  the  target  and  background 
C differential  =  (T-B).  In  other  applications,  it  may  be  normalized  to  be  dimensionless  and 
between  0  and  1,  Creiative  =  (T-B)/(T+B).  This  relative  contrast  is  particularly  common  in 
display  applications,  partly  because  it  reflects  some  features  of  human  vision.  The  eye 
adapts  to  the  overall  illumination  of  a  scene  so  that  the  relevant  parameterization  is  the 
relative  contrast.  One  speaks  of  a  target  that  is  10  percent  brighter  than  the  background, 
for  example.  One  may  also  see  Caitemative =  (T-B)/B. 
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Resolution  and  Sensitivity 


•  Resolution  and  sensitivity  are  the  basic  elements  of  a  sensor  description. 

•  Resolution  has  many  definitions  depending  on  what  task  is  to  be  done,  but 
they  are  usually  related  to  being  able  to  make  distinctions  about  the  target. 

-  Rayleigh  resolution  criterion:  Separating  two  images  of  a  star  (point  target) 

Unresolved  ik  Resolved  ★★ 

-  Bar  targets:  what  is  the  smallest  group  of  bars  that  can  be  resolved? 


Resolution  depends  on  the  image  contrast,  which  may  be  limited  by  the 
sensor’s  sensitivity:  the  signal  must  be  greater  than  the  noise. 


There  are  many  different  definitions  of  resolution,  each  of  which  is  connected  to  a 
specific  task.  For  example,  mathematically,  the  Rayleigh  criterion  requires  the  signal 
from  one  point  (star)  to  fall  to  zero.  Two  stars  are  resolved  when  one  star  is  located  at  the 
first  zero  point  of  the  other.  The  Sparrow  criterion  for  stars  only  requires  that  there  be  a 
dip  in  the  brightness  of  the  combined  image;  so  two  stars  may  be  resolved  on  the  Sparrow 
criterion,  but  not  on  the  Rayleigh  criterion. 

Although  the  use  of  a  calibration  target  with  bar  groups  of  different  sizes  is 
relatively  standard,  the  results  of  a  test  will  depend  on  the  number  of  bars  in  a  bar  group, 
the  aspect  ratio  of  the  bars,  and  the  instructions  to  the  test  subject  explaining  the  criterion 
for  declaring  a  bar  group  resolved. 
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Types  of  Digital  Systems 

EO/IR  digital  systems  can  be  divided  into  scanning  and  staring 
systems. 

Scanners  use  a  linear  array  of  detectors  (1  to  a  few  rows).  The 
image  is  developed  by  scanning  the  linear  array  across  the 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

□L 


scene. 

-  The  number  of  rows  of  detectors  may  be  >  1  to  provide: 

•  Staggered  phasing  to  reduce  sampling  artifacts  (aliasing) 

•  Additional  signal  gathering 

•  Multiple  wavelength  bands 

-  2nd  generation  FLIRs  used  a  vertical  array  of  detectors 
scanned  horizontally  using  scan  mirrors. 

-  Pushbroom  line  scanners  use  linear  arrays  that  are 
scanned  by  the  motion  of  the  aircraft. 

Starers  use  a  rectangular  array  of  detectors  that  capture 
an  entire  image  at  time.  □  □ 

-  Modern  systems  (both  EO  and  IR)  are  moving  to  staring  Q  Q 

arrays.  Q  □ 

-  Limitations  include  relatively  small  field  of  view,  j  |  [  j 

complexity  of  multi-band  systems,  and  aliasing. 

Behind  the  detectors  are  the  read-out  circuits. 

-  Readout  time  determines  “frame  rate”:  how  many  images/sec. 

-  Can  also  limit  the  integration  time,  sensitivity. 


Historically,  the  difficulty  in  manufacturing  large  square  or  rectangular  focal  plane 
arrays  has  meant  that  many  sensors  used  scanning  designs.  There  is  a  general  migration 
from  scanning  to  staring  sensors.  However,  in  some  cases,  a  line-scanner  may  still 
provide  the  best  solution.  For  example,  if  one  wants  a  number  of  different  colors  or 
bands,  a  line-scanner  can  simply  add  more  rows  of  detectors  for  the  additional  colors. 
Although  some  technologies  provide,  for  example,  a  two-color  IR  array,  achieving  four 
or  more  colors  requires  a  complicated  design  with  multiple  staring  arrays.  For 
applications  requiring  a  wide  field  of  view,  it  may  be  easier  to  create  a  10,000-element 
linear  array  for  a  line-scanner  than  a  1 -million  element  detector  staring  array. 
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Basics  Summary 

•  EO/IR  systems  cover  a  wide  range  of  distinct  technologies  based  on  the  targets 
and  the  mission  to  be  accomplished.  The  target  phenomenology  may  dominate 
the  choice  of  spectral  band.  For  example,  missile  launch  detection  depends  on 
the  very  hot  missile  exhaust,  which  produces  the  majority  of  radiation  in  the 
UV.  The  band  choice  is  also  influenced  by  the  vagaries  of  the  atmosphere 
(transmission  and  scattering). 

•  EO/IR  missions  divide  roughly  into  dealing  with  point  targets  and  fully  imaged 
targets.  For  point  targets,  the  challenge  is  to  extract  the  target  from  a  complex 
clutter  background.  For  fully  imaged  targets  (images  of  tanks,  for  example),  the 
contrast  between  the  target  and  the  background  is  a  central  parameter.  The 
ability  to  detect  dim  targets,  either  point  or  imaged,  is  a  measure  of  the  system’s 
sensitivity. 

•  The  resolution  of  a  sensor  is  a  measure  of  its  ability  to  determine  fine  detail. 
Measures  of  resolution  depend  on  the  precise  task.  The  use  of  eye-chart-like 
calibration  targets  is  common  in  DoD  applications. 

•  The  technology  used  in  EO/IR  sensors  may  be  divided  into  scanning  sensors, 
using  a  limited  number  of  detectors  that  scan  across  the  scene,  and  staring 
sensors,  using  large  numbers  of  detectors  in  rectangular  arrays. 
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B.  Non-Imaging  EO/IR  Systems 


Non-Imaging  EO/IR  systems 


Non-imaging  systems  refer  to  unresolved  or  point  targets. 


-  The  simplest  example  of  a  point  target  is  a  star:  in  the  sensor,  a  star 
produces  an  image  that  is  called  the  point  target  response,  impulse 
response,  or  blur  circle.  It  may  cover  more  than  one  detector  element 

-  An  unresolved  target  is  any  object  that  produces  an  image  in  the  sensor  /J? 
comparable  to  that  of  a  star:  there  is  no  internal  detail. 

-  For  a  resolved  target,  some  amount  of  internal  detail  can  be  recovered. 

-  The  rule  of  thumb  is  that  a  target  is  resolved  if  the  sensor  can  put  ^ 
multiple  detector  fields  of  view  on  the  target;  unresolved  if  the  target 
is  entirely  contained  within  one  instantaneous  field  of  view  (IFOV). 

-  IFOV  =  detector  size  (d)  /  focal  length  (fl) 

For  a  point  target,  the  only  signature  characteristic  is  the 
amount  of  energy  emitted  in  a  specified  direction  or  radiant 
intensity  measured  in  watts/ster  or  spectrally  (as  a  function 
of  wavelength)  watts/ster/pm 


L 

A 


-  Radiant  intensity  ~  Area  target  *  radiance  =  Af  *  l_T 

-  I_T  is  the  power  per  unit  solid  angle  per  unit  projected  source  area. 

-  Radiant  intensities  for  small  aircraft  (UAV)  are  the  range  of  0.05  watts/ster 
in  MWIR  and  0.25  watts/ster  in  LWIR. 


-  For  ballistic  missiles  the  signatures  are  much  higher,  megawatts/ster. 
Radiant  intensity  of  the  target  is  compared  to  that  of  the  background: 
radiant  intensity  contrast  =  Af  *(  LT  -  LBackground  ) 


Instead  of  using  watts/ster,4  it  is  sometimes  more  convenient  to  describe  the  photon 
flux,  photons/sec/ster.  The  common  unit  for  IR  systems  is  the  watt,  but,  for  visible 
systems,  photons/sec  is  often  employed. 

The  instantaneous  field  of  view  (IFOV)  is  a  fundamental  sensor  parameter.  At  range 
R,  a  single  IFOV  covers  a  distance  D  =  IFOV*R.  As  an  example,  detectors  for  visible 
digital  cameras  (Nikon,  Canon)  are  about  10  pm  in  size.  With  a  100-mm  lens  the  IFOV  = 
(10*10'6)/(100*10‘3)  =  10-4.  At  a  range  of  1,000  meters,  the  IFOV  covers  10  cm.  If  the 
target  were  1  meter  x  1  meter,  we  might  say  there  were  10  x  10  =  100  IFOVs  on  target; 
this  is  usually  replaced  with  the  sloppier  phrase,  1 00  pixels  on  target. 


4 


The  steradian  is  a  unit  of  solid  angle.  It  is  used  to  quantify  two-dimensional  angular  spans  in  three- 
dimensional  space,  analogously  to  how  the  radian  quantifies  angles  in  a  plane,  (wiki) 
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Non-Imaging  Applications 

•  Air-to-air  (Sidewinder)  —  First  Sidewinders  had  a  sing 
detector  behind  a  rotating  reticle.  Modern  air-to-air 
missiles  use  focal  plane  arrays  but,  at  long  range,  the 
target  is  a  point  target. 

•  IRST  (Infrared  Search  and  Track) 

-  Air-to-Air  —  Permits  passive  operation  of  aircraft.  Limited  by  cloud  clutter 
background. 

-  Surface  (Sea)-to-Air  —  Radar  detection  of  sea-skimming  cruise  missiles  may 
be  adversely  affected  by  environment  effects.  IR  has  different  environmental 
problems  and  may  provide  a  supplement  to  radar. 

-  Surface  (Ground)-to-Air  —  For  identification  and  tracking  of  small  aircraft 
(UAV)  EO/IR  may  offer  advantages  in  resolution. 

•  Missile  launch  detection 

-  Usually  based  on  missile  plume  detection 

-  Tactical  warning  systems 

-  Strategic  warning  (DSP/SBIRS) 


The  reticle  was  an  engineering  marvel.  A  reticle  rotating  disk  has  a  pattern  of 
transparent  and  opaque  sectors.  For  a  point  target,  the  rotating  pattern  provided  a 
blinking  signal  that  could  be  distinguished  from  the  background  (clouds  or  sky),  which 
did  not  vary  as  much.  Further  processing  of  the  shape  of  the  blink  actually  provided 
sufficient  information  to  guide  the  missile  to  the  target,  directing  it  to  maneuver  up, 
down,  left,  or  right. 

However,  the  problems  of  clutter  rejection  and  dependence  on  atmospheric 
conditions  are  barriers  to  IRST  use. 
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At  the  Sensor 

•  At  the  source,  the  radiant  intensity  contrast  is 

ARI  =  Aj (Ttarget  "^background  )  =  A?AL 

•  Neglecting  atmospheric  transmission,  the  total  signal  (watts  or 
photons/sec)  at  the  sensor  wijl  be 

Signal/sec  =  ARI 
4  ttR2 

where  Aoptics  is  the  area  of  the  sensor  lens  and  R  is  the  range 

•  Including  the  effects  of  transmission 

Signal/sec  =  r(A,R){Lbtga{A) ~  W-d f  r(A,R)ARI  {A)dA 

•  The  irradiance  of  the  target  (measured  in  watts/cm2  or 
photons/sec/cm2)  is  defined  by 

Irradiance  =  — — r-  f "  r(A,R)ARI  (A)dA 
4nR 

•  The  sensor  integrates  this  signal  over  a  time,  T,„f. 

Signal  =  AopticsTimIrradiance 


For  any  sensor  to  be  evaluated,  the  first  step  in  understanding  the  expected 
performance  is  to  understand  the  signal  produced  by  the  target.  The  signal  provided  by  a 
point  target  is  particularly  easy  to  understand.  The  example  introduces  several  generally 
useful  concepts.  The  radiant  intensity  contrast  is  a  characteristic  of  the  target  at  the 
source;  using  the  photon  (rather  than  watt)  description,  it  represents  the  number  of 
additional  photons/sec/ster  being  radiated  (or  reflected)  because  the  target  is  there. 

ARI  =  A.j{htarge^  —  L background )  =  AjAL 

The  number  of  these  photons  that  arrive  at  the  sensor  is  determined  by  the  range  and 
the  area  of  the  optics. 

photons / sec  =  ARI 

To  separate  out  sensor  characteristics  from  the  target  and  environment,  this  is 
rewritten  as 


photons /sec  =  A  opticsIr  radiance 


The  vast  majority  of  sensors  add  up  the  photons  received  over  some  integration  time 
so  that  the  signal  finally  is 

Signal  =  Number  of  photons  =  AopticsTintIrradiance 
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In  the  Sensor 


•  The  signal  is  not  necessarily  captured  by  a  single  detector, 
even  for  a  point  target. 


If  (most)  of  the  received  energy  lies  within  a  single 
detector,  it  cannot  be  located  more  precisely  than  !4  a 
detector  width. 


-  If  the  energy  is  spread  out  over  several  detectors,  the 
location  can  be  made  to  a  small  fraction  of  the  detector 
size  if  there  is  sufficient  signal-to-noise  ratio  (SNR). 

-  The  “ensquared  energy”  is  the  fraction  falling  on  the 
primary  detector  (typical  designs  are  for  ensquared 
energy  «  1/3). 

•  Finding  point  targets  is  easy  if  the  target  is 
bright  and  viewed  against  a  dim  background 
(stars  against  a  dark  sky):  this  is  not  usually  the 
case. 


-  Non-imaging  systems  employ  detection  algorithms  of  varying  complexity 
to  extract  possible  point  targets  from  the  scene. 

-  All  such  algorithms  have  both  a  probability  of  detection  (pd)  and  a  false 
alarm  rate  (FAR). 

-  The  performance  of  these  algorithms  depends  on  the  SNR,  the  difficulties 
of  the  environment,  and  the  detector  performance  characteristics. 


The  analysis  of  the  signal  given  on  the  preceding  chart  implicitly  assumed  that  all 
the  photons  from  the  target  were  easily  counted.  This  would  be  the  case  if  all  the  photons 
landed  on  a  single  detector.  However,  even  if  the  point  target  image  (or  “blur  circle”)  is 
small  compared  to  the  detector,  the  spot  may  land  on  the  boundary  between  detectors  so  a 
fraction  of  the  signal  falls  on  one  detector  and  another  fraction  on  another.  In  the  worst 
case,  the  signal  would  be  reduced  to  %  of  the  nominal  value,  divided  among  four 
detectors.  The  detection  algorithms  used  in  this  limit  may  look  at  the  detector  output  of 
several  detectors  to  increase  the  detection  range. 

In  the  other  limit,  the  size  of  the  blur  circle  may  be  large  compared  to  the  detector. 
Surprisingly,  this  can  be  an  advantage  in  some  point  target  applications  if  the  expected 
signal  level  is  high  enough.  For  a  tiny  blur  circle,  one  can  only  say  that  the  target  is 
located  somewhere  in  the  field  of  view  of  that  detector.  If  the  blur  circle  covers  several 
detectors,  a  model  of  how  the  signal  should  vary  from  detector  to  detector  can  allow  an 
estimate  of  the  location  of  the  target  to  a  fraction  of  an  IFOV.  This  is  also  called  “sub¬ 
pixel  localization.” 

In  most  applications,  the  detection  algorithms  are  as  important  in  determining 
performance  as  the  sensitivity  of  the  detectors  and  IFOV.  The  point  targets  may  be  seen 
against  a  complicated  clutter  background;  simply  looking  for  a  hot  spot  may  produce  an 
unacceptable  number  of  false  alarms. 
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Noise 


•  The  noise  associated  with  a  single  detector  is  relatively 
straightforward. 

-  Noise  due  to  the  average  background 

-  “clutter  noise”  may  be  incorporated  into  the  description 

-  Other  noise 

•  Background  noise  a  AT 

-  In  the  absence  of  a  target,  photons  still  N  =  detector  °Ptws  int  t 

arrive  on  the  detector.  On  average  (using  7rjt 

the  photon  flux  form  of  the  radiance). 

-  This  number  will  vary  statistically  even  for  a  constant  background, 
providing  a  contribution  to  the  noise  of  Nave1/2 

-  If  this  is  the  primary  contribution  to  the  noise,  the  sensor  is  called 
background  limited. 

•  Clutter  noise  —  In  many  cases,  the  variation  of  the  background  can 

be  incorporated  into  the  noise.  a  A  T 

If  this  is  dominant,  the  sensor  is  =  detector  °Ptlcs  int  xt 

Clutter  limited.  dutter  rfl1  background 

•  Adding  in  any  other  noise,  the  total  noise  isNoise2  =  Nave  +  5N]lutter  +  Other 2 


In  EO/IR  sensors,  estimating  and  characterizing  noise  and  clutter  is  a  vital  element 
for  understanding  system  performance.  The  use  of  a  “clutter-noise”  term  is  a  common 
first  order  attempt  to  incorporate  the  clutter  into  performance  estimates.  However,  since 
the  scene  is  processed  through  a  series  of  algorithms  (which,  in  some  applications  may  be 
quite  complex),  the  effect  of  clutter  has  to  be  considered  after  the  algorithms  are  run,  not 
before.  Estimates  of  performance  require  substantial  testing  or  laboratory  testing  with 
scene  simulators,  the  use  of  which  must  be  examined  carefully  to  ensure  that  the 
simulator  presents  the  relevant  features  of  the  clutter.  What  sort  of  clutter  is  relevant 
depends  on  the  algorithms! 

To  be  more  precise,  the  Lbackground  should  include  not  only  the  background  radiation 
at  the  target  but  also  the  so-called  “path  radiance,”  radiation  emitted  between  target  and 
sensor  by  the  atmosphere. 

Lfotai  =  L background  (at  target)  +  path  radiance  ( between  target  and  sensor) 
This  issue  arises  again  on  pages  28  and  35. 
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Noise  Equivalent  Irradiance  (NEI) 


•  For  simplicity,  assume  all  the  signal  is  received  on  a  single 
detector.  Then  the  signal-to-noise  ratio  would  be  given  by 


A, Irradiance 

C<~\T1D  _  optics  int 

Noise 

•  The  irradiance  that  would  provide  an  SNR=1  is  called  the  noise 
equivalent  irradiance. 


NEI 


photon 


Noise 
A  T 

i optics  int 


NEIwatts  = 


Noise  he 
A  T  A 

> optics  int  L sensor 


•  Note  that  the  relationship  between  NEI  and  the  system 

parameters  depends  on  whether  the  system  is  background  or 
clutter  limited. 


*  The  actual  signal  and  noise  at  the  output  of  the  algorithm  used 
for  target  detection  will  differ  from  the  signal  and  noise  used 
here. 


Noise  equivalent  irradiance  (NEI)  is  an  example  of  a  number  of  similar  figures  of 
merit.  For  example,  the  irradiance  of  a  target  depends  on  the  difference  in  the  radiance  of 
the  target  and  background.  For  a  system  that  utilizes  the  target’s  temperature  and  its 
emitted  radiation,  the  radiance  difference  can  be  considered  to  be  attributable  to  the 
difference  in  temperature  between  target  and  background.  If  the  temperature  difference  is 
relatively  small,  the  irradiance  will  be  proportional  to  the  temperature  difference,  AT. 
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Dealing  with  the  Environment 

•  Spectral  band 

-  For  some  applications  (detecting  and  tracking  aircraft),  one 
chooses  a  spectral  band  with  good  transmission. 

-  For  some  applications  with  very  bright  targets  (missile  launch 
detection),  the  clutter  problem  is  so  severe  that  a  band  might  be 
chosen  with  poor  transmission  to  suppress  background  clutter. 

•  Algorithms  for  extracting  the  target  from  the  background  may 
be  spatial  (essentially  suppressing  low  spatial  frequencies), 
temporal  (subtracting  two  images  to  cancel  non-moving 
clutter),  or  spatial-temporal  combinations. 

-  Simple  algorithms  may  have  large  false  alarm  rates.  For  example, 
cloud  edges  may  produce  false  targets  with  simple  spatial 
algorithms. 


EO/IR  systems  are  often  dominated  by  environmental  effects.  Normally  one 
chooses  a  band  with  good  transmission,  but,  with  good  transmission,  one  can  see  all  the 
clutter  as  well  as  the  target.  By  choosing  a  band  with  poor  transmission,  very  bright 
targets  can  still  be  seen  but  the  clutter  becomes  invisible.  If  this  trick  isn’t  used,  the 
targets  must  be  separated  from  the  background  by  means  of  detection  algorithms.  Any 
detection  algorithm  is  associated  with  both  a  probability  of  detection  (Pa)  and  a  false 
alarm  rate  (FAR).  Typically,  internal  parameters,  or  “thresholds”  in  the  algorithm,  permit 
the  designer  to  trade  off  probability  of  detection  and  false  alarm  rate.  The  curve  relating 
the  two  is  termed  the  “ROC  curve”;  ROC  is  an  acronym  of  ancient  origin  for  “receiver 
operating  characteristic.”  Testing  and  evaluating  the  detection  algorithm  is  difficult 
because  of  the  wide  range  of  potential  clutter  backgrounds.  Some  systems  may  employ 
scene  simulators  that  use  models  of  background  clutter  to  produce  a  wide  range  of  scenes 
purported  to  span  the  circumstances  of  the  real  system  in  the  real  environment. 
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Point  Target  Summary 

•  Non-imaging,  point  target  EO/IR  systems  focus  on  the  task  of  detecting  targets 
at  long  ranges.  For  these  applications,  the  details  of  the  target  are  irrelevant;  for 
example,  for  IR  sensors,  only  the  total  energy  emitted  matters,  not  the  precise 
temperature  and  size  of  the  target  separately.  The  energy  available  can  vary 
over  many  orders  of  magnitude,  from  the  engine  of  a  small  UAV  to  an 
intercontinental  ballistic  missile  (ICBM)  launch. 

•  Comparison  of  the  available  energy  at  the  sensor  to  the  noise  level  of  the  sensor 
provides  the  central  metric  of  sensor  performance,  the  noise  equivalent 
irradiance  or  NEI. 

•  The  problem  of  extracting  the  target  from  the  background  clutter  is  addressed  in 
two  basic  ways:  through  the  choice  of  sensor  band  and  algorithms.  For  missile 
launch  detection,  for  example,  one  may  use  the  “solar  blind”  UV  band.  The 
transmission  in  this  band  is  very  poor  and  the  general  background  is  reduced  to  a 
level  of  near  invisibility.  However,  the  very  bright  missile  signature  is  strong 
enough  to  still  provide  a  useable  signal.  Algorithms  for  separating  the  point 
target  from  the  structured  background  are  a  continuing  source  of  improvement 
in  performance,  but  their  complexity  may  make  the  evaluation  challenging. 
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TJTT 


Sampling  or  Aliasing 


Noise 


The  choice  of  optics  and  detector  affect  the  magnification  provided  and  image 
clarity.  Image  quality  includes  measures  of: 

•  Contrast  -  Degree  of  difference  between  lightest  and  darkest  portions  of  image 

•  Luminance  -  Brightness  of  image 

•  Noise  -  Random  signal  from  sources  outside  the  image  itself 

•  Sampling  -  Digitization  due  to  binning  of  signal  into  pixels 

•  Blur  -  Smearing  of  image  due  to  diffraction  and/or  imperfect  focus  (e.g.,  due  to 
jitter). 
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Modulation  Transfer  Function  (MTF) 


•  The  MTF  is  a  mathematical  description  of  image  blur:  the  MTF  is  the 
Fourier  transform  of  the  image  of  a  point  source. 

•  Many  components  of  the  optical  system  can  be  represented  by  individual 
MTFs  representing  the  impulse  response  of  each  part. 


•  Total  system  MTF  is  a  product  of  the  individual  MTFs  that  describe  each 
component  of  the  sensing 

MT^system  —  MTFoptjcs  MTFdetector  MTFatmoSphere  MTF^jspiay  MTFother 


•  If  the  MTF  is  known  (or  equivalently,  the  blur  function)  one  can  convolve 
the  original  scene  with  the  blur  to  determine  the  effect  of  the  optics  on  the 
image 


The  ability  to  calculate  or  estimate  the  different  Modulation  Transfer  Function 
(MTF)  factors  varies.  For  perfect  or  “diffraction  limited”  optics,  the  MTF  has  a  well- 
known  analytic  expression.  For  realistic  optics,  the  different  aberrations  that  reduce  the 
MTF  can  be  modeled  well;  computer-based  ray-tracing  programs  can  predict  relatively 
accurately  the  overall  optical  MTF.  The  detector  and  display  can  be  modeled  accurately 
as  well.  However,  the  atmospheric  effects  are  represented  by  statistical  models  and  are  a 
source  of  uncertainty.  For  any  specific  system,  the  “other”  effects  (including,  for 
example,  focus  error)  will  often  be  represented  empirically. 
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Modulation  Transfer  Function  (MTF) 

•  The  Modulation  Transfer  Function  (MTF)  is  the  single-most 
important  characteristic  of  an  optical  system. 

•  The  MTF  is  simply  the  frequency  response  of  the  system: 

-  The  image  of  a  target  can  be  decomposed  into  sinusoids  of  different 
frequencies;  knowing  how  the  system  responds  to  each 
frequency  separately  characterizes  the  system. 

-  If  the  signal  has  a  component  of  A  sin(2pfx)  and  the  response  is 
0.8  *  A  sin(2pfx),  then  the  MTF(f)=0.8. 

•  In  general,  the  wider  the  MTF(f),  the  better  the  resolution 

-  The  blur  circle  or  impulse  response  is  the  Fourier  transform  of  the 
MTF(f):  wide  MTF  <=>narrow  blur  circle. 

-  A  rule  of  thumb  is  that  the  resolution  limit  of  the  system  is  at  the 
frequency  for  which  MTF(f)  =  0.1 . 

-  Blur  circle  is  short-hand  for  the  shape  of  the  impulse  response: 
there  is  typically  a  smooth  oscillating  form. 

•  Every  element  of  the  total  system  can  affect  the  MTF 

-  Optics  themselves 

-  Film  grain  or  detector  size 

-  Vibration 

-  Atmospheric  distortions 

-  Electronics  including  display 

-  Human  eye! 
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Optics  and  Detector  MTF 


•  The  primary  contributors  to  the  MTF  curve  in  modern  systems  are  the 
optics  MTF  and  the  detector  MTF. 


•  The  ideal,  or  diffraction  limited,  optics  MTF 
establishes  a  maximum  spatial  frequency  of 

-  D/A  (measuring  in  cycles/radian)  or 

-  1/Af#  (measuring  on  the  focal  plane  in 
cycles/m) 

where  D  is  the  diameter  of  the  optics  and 
f#  is  the  f-number,  f#=focal  length/  D. 

-  The  diffraction-limited  MTF  is  nearly  a  straight 
line,  but  lens  aberrations  will  reduce  the  MTF. 

•  The  ideal  (square)  detector  MTF  =  sinc(pfd) 

-  Sinc(x)=sin(x)/x 

-  d=  detector  size 

-  Zeros  of  detector  MTF  =  n/d,  n  =  1,  2,... 

•  System  design  will  determine  relative 
location  of  detector  MTF  zero  and  optics 
MTF  zero. 

-  Optics  are  typically  “better”  Af#  <  d. 


-  This  can  lead  to  sampling  artifacts  or  “aliasing.” 


Classical  MTF  measurements  in  the  laboratory  are  made  more  complicated  by 
aliasing  effects.  One  approach  to  overcome  aliasing  is  to  displace  the  image  of  the 
calibration  target  to  provide  different  sample  phasing  (see  p.60).  For  examples  of  other 
efforts  made  to  work  around  the  sampling  artifacts,  see  Research  Technology 
Organization  Technical  Report  75(11),  Experimental  Assessment  Parameters  and 
Procedures  for  Characterization  of  Advanced  Thermal  Imagers. 
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System  Design 


•  Standard  staring  focal  planes  always  the  potential  for 
aliasing. 

-  A  system  is  not  aliased  if  the  maximum  frequency  is  less 
than  half  the  sampling  frequency  =  fNyquist. 

-  For  detectors  of  size  d,  for  staring  arrays  the  sample 
frequency  is  1/d  so  the  maximum  frequency  should  be 

^  fNyquist  “  1/2d. 

-  But  the  first  zero  of  the  detector  MTF  =  1/d. 


By  degrading  the  optics  (increasing  the  size  of  the 
blur  circle),  contrast 
is  lost  but  aliasing  is 
reduced. 

This  major  system 
trade  will  be 
discussed  in 
detail  in  Section  F. 
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Aliasing  refers  to  the  inability  to  correctly  represent  spatial  frequencies  higher  than 
the  Nyquist  frequency  of  a  digital  system.  This  will  be  discussed  in  detail  in  Section  F. 

This  chart  provides  an  initial  discussion  of  why  modem  staring  systems  are  typically 
aliased.  For  a  staring  array  of  square  detectors,  the  detector  MTF  is  given  by 

MTFdet(f )  =  sincinfW )  = 

The  detector  MTF  therefore  extends  to  all  frequencies.  The  first  zero  of  the  detector 
MTF  is  at  f  =  1/d;  subsequent  zeros  are  at  multiples  of  that  frequency,  f  =  m/Wd  m=  1,  2, 
3.... 


However,  the  Nyquist  frequency  for  such  an  array  is  fNyquist  =  l/2d.  Even  if  the 
frequencies  beyond  the  first  zero  of  the  MTF  are  ignored  (because  of  their  diminishing 
amplitudes),  the  system  is  under-sampled  by  a  factor  of  2. 

This  may  be  modified  by  the  consideration  of  the  optics  MTF,  represented  here  by 
straight  lines  (correct  for  a  square  not  circular  lens!).  If  the  blur  circle  of  the  detector 
(first  zero)  fits  inside  the  detector,  the  optics  MTF  extends  to  2/d  (and  is  zero  for  high 
frequencies).  This  would  mean  being  four  times  under-sampled  (maximum  frequency  4  x 
Nyquist).  Degrading  the  optics  until  the  blur  occupies  four  detectors  leads  to  the 
“matched  optics”  condition  for  which  the  optics  MTF  zero  coincides  with  the  first 
detector  zero.  Only  if  the  blur  occupies  16  detectors  does  the  optics  MTF  zero  match  the 
Nyquist  limit. 

As  shown  in  the  preceding  examples  and  to  be  developed  further,  this  elimination  of 
all  potential  aliasing  reduces  the  system  MTF  (contrast)  significantly.  Most  design  trades 
permit  some  aliasing  in  order  to  preserve  useable  image  contrast. 
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Minimum  Resolvable  Temperature 

•  Minimum  Resolvable  Temperature  (MRT)  is  a  system  performance 
measure  developed  in  the  1970s  to  balance  two  system  characteristics 

-  Sensitivity,  measured,  for  example,  by  the  noise  equivalent  temperature 
difference  (NETD) 

•  This  was  defined  as  the  difference  of  the  temperature  between  a  (large)  object 
and  the  background  required  to  have  a  signal  that  produced  a  signal-to-noise 
ratio  (SNR)  of  unity:  SNR  =  1 

-  Resolution,  either  measured  in  terms  of  the  system  modulation  transfer 
function  (MTF)  or  the  size  of  a  detector  instantaneous  field  of  view  (IFOV) 

•  For  example,  resolution  defined  as  the  spatial  frequency,  f,  at  which  the  MTF(f)  = 
0.1 

•  IFOV  =  detector  size  /  focal  length 

•  MRT  is  defined  in  the  laboratory  with  a  4-bar  target 

-  Bar  in  7:1  ratio  so  whole  target  represents  a  square 

-  The  size  of  the  target  and  the  laboratory  optics  defined 
a  specific  spatial  frequency  as  seen  by  the  device,  f. 

-  The  4  bars  are  maintained  at  a  specified  temperature 
above  the  background  and  inner  bars. 

-  The  temperature  of  the  bars  is  lowered  until  the  technician  can  no  longer 
discern  all  four  bars. 

-  This  temperature  was  the  minimum  resolvable  temperature  MRT(f). 


In  classical  noise  equivalent  temperature  difference  (NETD),  the  signal  was 
compared  to  a  measure  of  the  temporal  noise  of  the  system  (as  measured  through  a 
standard  electronic  filter).  In  modem  staring  focal  plane  arrays,  other  noise  sources  must 
be  considered.  For  example,  each  detector  in  the  array  will  generally  have  a  different 
response;  this  is  called  detector  non-uniformity.  Although  an  effort  is  made  to  provide 
non-uniformity  correction,  any  residual  non-uniformity  represents  a  static  spatial  pattern 
noise.  In  addition,  individual  detectors  may  be  uncorrectable,  noisy,  or  dead. 

The  NVESD  has  introduced  a  three  dimensional  (3D)  noise  model  to  describe  noise 
components  in  the  temporal  and  two  spatial  dimensions. 
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MTF/MRT  and  Sampling  Artifacts 


•  The  sampling  artifacts  of  modern  staring  focal  plane 
arrays  make  the  definitions  of  the  MTF  (and  hence, 
MRT)  somewhat  problematic  for  modern  systems. 

•  The  first  image  shows  a  “well-sampled”  image. 

-  Depending  on  the  human  observer,  the  smallest  pattern 
that  is  resolved  (counting  4  bars)  is  one  of  the  two 
circled  bar  groups. 

-  This  sensor  will  have  an  MTF  (and  MRT)  that  can  be 
measured  by  classical  approaches 

•  The  second  image  is  “aliased”. 

-  The  bar  groups  are  distorted  and  only  the  largest  group 
is  resolved 

-  Modified  techniques  are  required  to  measure  (rather 
than  model)  the  MTF  and  MRT. 

•  Mission  performance  will  be  between  the  Nyquist  and 
un-aliased  limits.  See  discussion  in  section  on 
aliasing  and  notes  page  below. 


Well-sampled 


Distorted  images 
seem  to  have  only  3 
bars 


One  approach  taken  by  NVESD5  and  others  to  represent  the  effect  of  aliasing  on  the 
MTF  is  to  represent  the  MTF  of  the  sampled  system  by  squeezing  (scaling)  the  MTF  of 
the  corresponding  unsampled  system. 

The  factor  R  =  1  -  k  SR,  where  SR  is  a  metric  representing  the  spurious  response. 
The  value  of  the  constant  k  may  vary  with  task  (detection,  recognition,  identification)  and 
different  forms  of  the  SR  metric  have  been  used.  H(f)  is  the  Fourier  transform  of  the 
reconstruction  function  used.  See  Section  G  on  aliasing. 

Another  approach  is  to  treat  aliasing  as  a  noise  term.6 

An  extension  of  the  MRT  task,  the  Minimum  Temperature  Difference  Perceived 
(MTDP)  has  been  introduced  in  Germany.  This  provides  an  extension  beyond  Nyquist. 


5  Driggers,  Vollmerhausen,  and  O’Kane,  SPIE,  Vol.  3701,  pp.  61-73,  1999. 

6  Vollmerhausen,  Driggers  and  Wilson,  JOSA-A,  Vol  25,  pp.  2055-2065,  2008. 
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Knobs 

*  From  a  user  perspective  and  a  testing  perspective,  controls  over  the 
digital  image  may  affect  performance  significantly. 

-  In  some  missions,  there  may  be  insufficient  time  to  explore  alternative  processing 
approaches. 

-  The  effects  of  alternative  processing  are  not  completely  modeled  so  performance  may  be 
better  or  worse  than  expected  from  standard  models. 

*  Basic  controls  include  gain  and  level,  but  many  options  could  be  made 
available  to  the  user/tester. 

*  “Photoshop”  operations 

-  Sharpening 

-  Histogram  manipulation 

-  Zooming  interpolation  (reconstruction) 

-  Color  balance 

*  Hidden  algorithms 

-  Producing  full  color  images  from  Bayer 
cameras 

-  Combining  images  from  multiple  lens  Original  Sharpened  &  equalized 

-  Combining  images  from  multiple  frames 

*  Red  items  will  be  discussed  more  completely  in  the  appendixes. 


Performance  metrics  can  be  affected  even  by  simple  gain  and  level  adjustments. 

Contrast  =  (Signal  -  Background)/!} ackground  =  (S-B)/B 

is  unaffected  by  a  change  of  scale  (gain)  but  is  changed  dramatically  with  a  level  shift: 
Contrast  =  (( S-L )  -  (B-L))/(B-L)=  (S-B)/(B-L). 

Signal-to-noise  (SNR)  =  (S-B)/Noise 

is  unaffected  by  gain  or  level  as  long  as  values  are  not  truncated  or  saturated. 

In  some  applications,  gain  and  level  may  not  be  under  user  control. 
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Key  Concepts  for  Image  Quality  Modeling 

•  Spatial  frequency  of  image  content 


low  frequency 

•  Contrast  -  shades  of  grey/color 


limn  I  in 

high  low  reducing  contrast 

*  Eye  Contrast  Threshold  Function  (CTF)  -  Ability  of  eye  to  discern  contrast 

•  Modulation  Transfer  Function  (MTF)  -  Degradation  to  image  contrast 
caused  by  aspects  of  collection  process 


high  frequency 


Considering  bar  targets,  as  in  the  discussion  of  minimum  resolvable  temperature 
(MRT),  increasing  spatial  frequency  (i.e.,  reducing  the  width  of  the  bars)  generally  has  a 
negative  effect  on  the  human  eye’s  ability  to  resolve  targets,  as  does  reducing  contrast 
(the  difference  in  brightness  between  light  and  dark  bars).  A  convenient  way  to  capture 
the  interaction  between  spatial  frequency  and  contrast  is  to  plot  the  eye  contrast  threshold 
function  (CTF)  in  a  graph  of  contrast  versus  spatial  frequency.  Such  a  plot  shows  that  as 
the  bars  become  narrower,  greater  contrast  is  required  to  distinguish  black  bars  from 
white  bars.7 

The  shape  of  the  modulation  transfer  function  in  the  contrast  versus  spatial 
frequency  plot  shows  that,  as  spatial  frequency  increases,  the  ability  of  a  system  to 
transfer  or  portray  contrast  differences  diminishes.  The  better  the  system  (including 
optics,  detector,  atmosphere,  and  display)  is,  the  shallower  is  the  drop  off  of  the  curve 
with  increasing  spatial  frequency. 


7  Interestingly,  at  very  low  spatial  frequencies,  more  contrast  is  required  to  distinguish  bars,  just  as  at 
higher  spatial  frequencies. 
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Evolution  of  Tactical 
Models  for  Predicting  Image  Quality 


•  Johnson  Criteria:  Started  with  experiments  conducted  by  John 

Johnson,  an  Army  Night  Vision  Laboratory  [later  renamed  Army  Night 
Vision  and  Electronic  Sensors  Directorate  (NVESD)]  scientist,  in  the 
1950s 


•  In  the  1950s,  sensors  were  analog  (film,  TV).  Johnson  Criteria 
had  to  be  translated  into  the  digital  sensor  world. 

•  Targeting  Task  Performance  (TTP):  Today  NVESD  maintains 
community-standard  models  that  predict  the  image  quality  and 
associated  probabilities  of  detection,  recognition,  and  identification 
for  various  vehicles  and  dismounts  with  differing  backgrounds  in  the 
visible  and  infrared  regions  of  the  electromagnetic  spectrum. 


Visible  -  Solid  State 
Camera  and  Image 
Processing  (SSCamlP) 
Performance  Model 


IR  -  Night  Vision  Thermal 
and  Image  Processing 
(NVThermlP)  Performance 
Model 


In  1957  and  1958,  John  Johnson  at  the  Army  Night  Vision  Laboratory  developed  a 
series  of  experiments  to  analyze  the  ability  of  observers  to  perform  visual  tasks,  such  as 
detection,  recognition,  and  identification.  He  came  up  with  general  rules  for  the  number 
of  bar  pairs  required  on  target  to  have  a  50-percent  probability  of  performing  each  of 
these  tasks.  Over  time,  this  approach  has  evolved  to  account  for  the  increasing 
sophistication  of  today’s  EO/IR  sensors.  The  Army  Night  Vision  and  Electronic  Sensors 
Directorate  (NVESD)  maintains  models  for  predicting  image  quality  based  on  the 
targeting  task  performance  (TTP)  metric,  the  successor  to  the  Johnson  criteria.  Very 
recently,  the  visible  and  IR  models,  known  as  Solid  State  Camera  and  Image  Processing 
(SSCamlP)  and  Night  Vision  Thermal  and  Image  Processing  (NVThermlP),  respectively, 
have  been  combined  into  one  software  package,  known  as  the  Night  Vision  Integrated 
Performance  Model  (NV-IPM). 
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Johnson  Criteria 


•  How  many  pixels  are  required  to  give  a  50%  probability  of  an  observer  discriminating 


an  object  to  a  specified  level? 

•  Experiments  with  observers  yielded  the  following 

-  Detect  (determine  if  an  object  is  present)  1 .5  pixels 

-  Recognize  (see  what  type  of  object  it  is;  e.g.,  person,  car,  truck)  6  pixels 

-  Identify  (determine  if  object  is  a  threat)  12  pixels 

•  These  are  the  number  of  pixels  that  must  subtend  the  critical  dimension  of  the  object, 
determined  by  statistical  analysis  of  the  observations 

-  Critical  dimension  of  human  0.75  m 

-  Critical  dimension  of  vehicle  2.3  m 


Hence  for  a  human,  the  requirements  are 

-  Detect  2  pixels/meter 

-  Recognize  8  pixels/meter 

-  Identify  16  pixels/meter 

For  a  man  who  is  1.8  m  x  0.5  m,  this  corresponds  to  requirements  of 

-  Detect  3.6  pixels  tall  by  1  pixel  wide 

-  Recognize  14.4  pixels  tall  by  4  pixels  wide 

-  Identify  28.8  pixels  tall  by  8  pixels  wide 


As  an  example,  the  Johnson  criteria  can  be  applied  to  determine  the  number  of 
pixels  in  an  image  required  to  detect,  recognize,  and  identify  a  human  subject.  The 
calculations  shown  here  are  illustrated  by  the  images  on  the  chart  below. 


Images  Illustrating  Johnson  Criteria 


Detect:  Something  is 
present. 


Recognize:  A  person 
is  present. 


Identify:  The  person 
is  holding  an  object 
that  may  pose  a 
threat. 


Source:  FLIR  Technical  Note  at  http://www.flir.com/uploadedFiles/ENG_01_howfar.pdf 
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Eye  Contrast  Threshold  Function  (CTF) 


•  One  of  the  most  common  and  useful  ways  of  characterizing  human  vision 

-  One-dimensional  sine  waves  are  displayed  to  observer. 

-  For  a  given  luminance  level  and  signal  spatial  frequency,  CTF  is  the  minimum 
perceivable  contrast  vs.  visual  acuity  (resolution). 

-  Derived  from  two-alternative  forced  choice  experiment:  observer  must  choose 
between  sine  wave  and  blank  field. 

-  Repeat  procedure  for  different  average  luminance  levels. 

-  Conduct  tests  at  varying  spatial  frequencies  to  derive  curves  shown  below. 


Spatial  Frequency  (cycles/mrad) 

*fL:  luminance  is  expressed  in  units  of  foot- Lamberts 


The  eye  contrast  threshold  function  has  been  measured  by  repeated  experimentation 
on  a  large  group  of  observers.  The  experiment  entails  showing  two  fields  to  the 
observers.  One  field  contains  alternating  black  and  white  bars,  and  the  second  is  constant 
gray  of  the  same  total  luminance.  The  observers  are  asked  to  identify  which  of  the 
images  contains  the  alternating  bars.  This  experiment  is  repeated  as  the  contrast  of  the 
bar  pattern  is  changed,  while  keeping  the  total  luminance  constant.  The  contrast  at  which 
75  percent  of  the  observers  correctly  identify  the  bar  pattern  is  considered  the  threshold 
for  that  level  of  luminance.  (The  threshold  is  set  at  75  percent  because  in  a  two- 
alternative  forced  choice  experiment  such  as  the  one  described  here,  the  right  answer  will 
be  selected  50  percent  of  the  time  simply  by  chance.)  By  conducting  this  experiment  at 
varying  spatial  frequencies  (i.e.,  bar  spacings),  one  of  the  curves  shown  above  can  be 
measured.  Differing  levels  of  luminance  give  rise  to  the  series  of  curves  shown  above. 
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How  Good  is  the  Human  Eye? 

Contrast  Option  A  Option  B 


For  each  row,  the  viewer  is  asked  which  option  contains  alternating  black  and 
white  lines  and  which  option  is  a  uniform  grey  bar. 

This  series  of  six  rows  demonstrates  that  it  becomes  more  difficult  to 
distinguish  features  as  the  contrast  is  decreased  and  as  the  features  become 
finer. 


An  example  of  the  experiment  described  in  the  previous  chart  is  displayed  here. 
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Approach  to  Measuring  Sensor  Effectiveness 

•  Night  Vision  Lab  model  (SSCAM,  NVTHERM)  estimates  of  target  detection/recognition/ 
ID  probability  based  on  the  TTP  (targeting  task  performance)  metric,  which  has 
supplanted  Johnson  criteria 

-  TTP  =  difference  between  target-background  contrast  and  the  eye’s  capability  to  distinguish 
contrast 

•  Summed  over  all  spatial  frequencies 

•  CTGT  is  a  function  of  range 

-  Experimental  data  are  “curve  fit”  based  on  the  TTP  value 


TTP  =  | 


target  contrast 
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Modulation  transfer  function 

(propagation,  sensor,  observation) 
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eve  contrast  threshold 
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Range  *V50 


^  IT, 
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Detection  probability  (similar  formula  for  recognition,  ID) 


CtgtMTF 

(observed  target  contrast)  /  CTF 

(eye  threshold) 

TTP 

(“usable”  contrast) 


Source:  “The  TTP  Metric,”  Vollmerhausen  and  Jacobs,  NVESD 


The  TTP  metric  used  by  NVESD  captures  the  difference  between  the  target- 
background  contrast  and  the  eye’s  ability  to  distinguish  that  contrast.  The  square  root 
functional  form  ensures  that  excessive  amounts  of  “usable”  contrast  are  not  given  too 
much  weight  in  the  metric.  The  probability  of  detection,  recognition,  or  identification  is 
related  to  the  TTP,  target  size,  target-sensor  range,  and  V50  through  the  target  transfer 
probability  function,  which  is  a  logistics  curve  with  an  exponent  determined  by  curve 
fitting.  The  task  difficulty  parameter,  V50,  is  analogous  to  the  Johnson  N50  value  that 
represents  the  number  of  resolvable  cycles  on  the  average  target  for  a  50-percent 
probability  of  detection/recognition/identification.  For  each  target  type  in  a  set  of  target 
options,  V50  is  determined  by  a  series  of  calibration  experiments  involving  human 
perception  of  the  original  image  and  images  degraded  by  a  known  amount. 
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Target  Contrast 


•  CTGT_0  is  the  zero  range  contrast  value  for  a  target  on  a  background. 

-  In  visible,  this  is  a  measure  of  the  difference  in  reflectivities  between  target  and 
background. 

-  e.g.,  contrast  for  individual  wearing  white  against  concrete  background  is  less  than 
contrast  for  individual  wearing  black  against  same  background. 

-  In  IR,  this  is  a  measure  of  the  difference  in  target  and  background  temperatures. 

•  The  apparent  target  contrast  at  a  distance  is  related  to  CTGT_0  via  the  extinction 
coefficient  p  from  Beer’s  Law. 

-  CTGT=  CTGT_0  e-Px  where  p  is  in  knr1  and  x  is  the  range  from  target  to  sensor  in  km. 

-  Does  not  account  for  imaging  through  non-homogeneous  media,  such  as  the 
atmosphere. 

•  Apparent  target  contrast  is  degraded  by  scattering  of  light  into  the  path  between  target 
and  sensor. 


This  depends  on  the  amount  of  aerosols  and  other  particulates  present,  as  well  as 
any  other  potential  sources  of  scatter,  and  it  is  particularly  sensitive  to  the  relative 
positions  of  the  sun,  target,  and  sensor. 

Army  uses  sky-to-ground  ratio  concept 
to  account  for  this. 


The  zero  range  target  contrast  is  a  measure  of  the  inherent  contrast  between  the 
target  and  the  background,  independent  of  the  distance  between  the  target  and  sensor.  In 
the  visible,  the  contrast  at  each  wavelength  is  integrated  over  the  visible  portion  of  the 
electromagnetic  spectrum  to  determine  the  zero  range  target  contrast.  In  the  infrared,  the 
contrast  is  due  to  the  difference  in  thermal  emission  between  target  and  background.  The 
zero-range  contrast  is  then  attenuated  by  atmospheric  effects,  which  can  be  treated  as 
homogeneous  or  modeled  in  a  complex  computer  program,  such  as  MODTRAN 
(MODerate  resolution  atmospheric  TRANsmission).  The  apparent  target  contrast  is  also 
degraded  by  scattering  of  light  into  the  path  between  target  and  sensor. 
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Sky-to-Ground  Ratio  (SGR) 

•  SGR  is  related  to  transmittance  t  via  T  = _ \ _ 

1  +  SGR{epx  - 1) 

•  SGR  is  a  measure  of  the  scattering  of  light  into  the  path  between 
target  and  sensor. 

•  In  clear  conditions  with  low  relative  humidity  where  direct  sunlight  is 
not  close  to  the  target-sensor  path,  a  value  of  1.4  may  be  used  for 
SGR. 

•  SGR  approaches  10  or  greater  when  direct  sunlight  is  near  the  target¬ 
sensor  path. 

•  Extinction  coefficients  as  a  function  of  wavelength  are  tabulated  for 
urban,  rural,  and  maritime  models. 

•  Combined  with  representative  SGRs,  this  allows  for  calculation  of 
transmittance  in  a  range  of  environmental  situations. 


The  Army  has  introduced  the  concept  of  sky-to-ground  ratio  (SGR)  to  account  for 
the  scattering  of  light  into  the  path  between  target  and  sensor.  Values  of  SGR  typically 
range  from  1.4  in  clear  conditions  with  low  relatively  humidity  to  well  over  10  when  the 
path  radiance  is  high,  due  to  multiple  scattering  events,  as  would  be  encountered  when 
the  sun  is  shining  directly  on  the  target-sensor  path  or  in  a  forest  under  overcast 
conditions. 
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National  Imagery  Interpretability  Rating  Scale  (NIIRS) 

*  Standard  used  by  the  strategic  intelligence  community  to  judge  the  quality 
of  an  image  (versus  the  Johnson  criteria/TTP  approach  used  by  the  tactical 
intelligence  community). 

*  NIIRS  of  particular  image  is  based  on  judgment  of  imagery  analysts,  guided 
by  updated  published  scale  of  representative  examples. 

-  e.g.,  NIIRS  =  5  means  can  identify  radar  as  vehicle-mounted  or  trailer-mounted 

*  NIIRS  can  be  estimated  from  General  Image  Quality  Equation,  derived  from 
curve  fitting. 


NIIRS  4  (=  6’  GSD) 


The  strategic  intelligence  community  uses  the  National  Imagery  Interpretability 
Rating  Scale  (NIIRS)  to  evaluate  the  quality  of  still  imagery.  NIIRS  is  a  qualitative 
scale,  running  from  0  to  9,  with  guidance  for  imagery  analysts  provided  in  the  form  of  a 
table  of  example  operations  that  could  be  performed  at  various  NIIRS  values.  The  table 
for  visible  NIIRS  values  is  provided  on  the  following  chart.  There  are  also  tables  for  IR 
NIIRS,  synthetic  aperture  radar  (SAR)  NIIRS,  civil  imagery  NIIRS,  multi-spectral 
NIIRS,  and  video  NIIRS. 

NIIRS  can  be  estimated  with  the  General  Image  Quality  Equation  (GIQE),  an 
empirical  model  that  has  been  designed  to  predict  NIIRS  from  sensor  system 
characteristics.  In  the  visible,  the  GIQE  takes  the  form 

NIIRS  =  10.251  -  a  log  GSD  +  b  log  RER-  0.656  H  -  0.344  (G/SNR) 

where  GSD  is  the  ground  sample  distance 
RER  is  the  relative  edge  response 
H  is  the  overshoot, 

G  is  the  gain, 

SNR  is  the  signal-to-noise  ratio, 
a  =  3.16 

b  =  2.817  for  RER  <  0.9  and  a  =  3.32,  b  =  1.559  for  RER  >  0.9. 

The  only  change  to  the  GIQE  in  the  IR  is  that  the  constant  term  is  10.751,  rather  than 
10.251.  The  GSD,  RER,  and  overshoot  will  be  defined  and  discussed  in  subsequent 
charts. 
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Images  with  Different  GSD 


GSD  =  5”  in  upper  left  image 

10”  in  upper  right  image 
2’  in  lower  left  image 
GSD  »  IFOV  *  range 

Original  5”  GSD  Image  Source: 
http://www.isprs.org/education/PDF/GOA_ 
David_HR_mapping_Section8.pdf 
Tutorial  by  David  Holland  et  al.  for  ISPRS 
meeting 

Subsequent  images  generated  in  MATLAB  via 
pixel  averaging 


Ground  sample  distance  (GSD)  is  the  dominant  term  in  the  GIQE  and  ultimately 
provides  an  upper  limit  to  the  image  quality,  since  all  information  in  the  area  defined  by 
the  horizontal  and  vertical  GSDs  is  incorporated  into  a  single  pixel.  The  GSD  is  defined 
by 

GSD  =  ((PP  *  x)/Fl)/cos  0 

where  Pp  is  the  pixel  pitch 
x  is  the  slant  range 
FL  is  the  focal  length 

0  is  the  look  angle,  measured  between  nadir  and  the  sight  line. 

For  non-square  pixels,  the  vertical  and  horizontal  GSDs  differ  and  are  specified 
separately. 

The  blurring  of  the  original  image  as  adjacent  pixels  are  averaged  together  is 
evident  in  comparing  the  image  in  the  upper  left  quadrant  to  that  in  the  upper  right 
quadrant  and  finally  to  that  in  the  lower  left  of  the  figure. 
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Relative  Edge  Response 


Sources:  Kim  et  al., 

Image-Based  Estimation 
and  Validation  of  Nil RS  for 
High-Resolution  Satellite 
Images,  2008. 

Blonski  et  al.,  Spatial  Resolution  Characterization  for 
Aerial  Digital  Imagery,  2006. 


Appearance  of  Image 


Distance  /  GSD 


The  relative  edge  response  is  a  measure  of  blur  and  is  defined  to  be  equal  to  the 
difference  in  the  normalized  edge  response  0.5  pixels  to  either  side  of  an  edge.  In  a 
perfect  image  where  a  completely  black  area  is  adjacent  to  a  completely  white  area,  the 
pixel  on  one  side  of  the  edge  has  a  luminance  of  one  whereas  the  pixel  on  the  other  side 
has  a  luminance  of  zero.  In  reality,  blurring  of  the  black  and  white  portions  occurs  in  the 
vicinity  of  the  edge.  Edge  analysis  is  conducted  on  real  images  to  ascertain  values  of 
relative  edge  response  (RER)  for  actual  sensors. 
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Overshoot 


Overshoot:  the  introduction  of  artifacts,  such 
as  halos,  through  excessive  use  of  a  digital 
process  to  make  images  appear  better  by 
making  edges  sharper 

Original  Image 


Sharpened  Image 


Observe  that  the  bottom  image 
has  been  “brightened”  by  a 
digital  process  (MTF  correction) 

Source:  Imatest 

(http://www.imatest.com/docs/MTF_appearance.html ) 


Overshoot  is  due  to  excessive  use  of  a  digital  process  to  sharpen  imagery  at  edges 
(MTF  compensation).  It  is  defined  to  be  the  value  of  the  peak  normalized  edge  response 
in  the  region  1-3  pixels  from  the  edge,  unless  the  edge  is  monotonically  increasing  in  that 
range,  in  which  case,  it  is  defined  as  the  edge  response  at  1 .25  pixels  from  the  edge.  This 
term  partially  offsets  the  improvement  in  NIIRS  that  a  really  sharp  edge  response 
provides. 
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Imaging  EO/IR  Systems  Summary 

•  The  effectiveness  of  imaging  systems  can  be  degraded  by  many  factors, 
including  limited  contrast  and  luminance,  the  presence  of  noise,  and  blurring 
due  to  fundamental  physical  effects. 

•  MTF  is  a  mathematical  description  of  image  blur  and  can  be  broken  into  each 
component  of  the  sensing,  such  as  optics,  detector,  atmosphere,  and  display, 
providing  insight  into  the  sources  and  magnitude  of  image  degradation. 

•  The  Army  NVESD  maintains  models  for  predicting  image  quality  based  on  the 
TTP  metric,  the  successor  to  the  Johnson  criteria,  rules  of  thumb  for  the  number 
of  bar  pairs  required  on  target  to  have  a  50-percent  probability  of  detecting, 
recognizing,  or  identifying  the  target. 

•  The  strategic  intelligence  community  uses  NIIRS  to  evaluate  the  quality  of  still 
imagery. 

-  NIIRS  is  a  qualitative  scale,  running  from  0  to  9. 

-  Tables  are  published  associating  rating  levels  with  tasks  for  EO,  IR,  SAR, 
civil  imagery,  multi-spectral,  and  video. 

-  NIIRS  can  be  estimated  by  the  GIQE,  an  empirical  model  that  has  been 
designed  to  predict  NIIRS  from  sensor  system  characteristics. 
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D.  Tracking 


Tracking  Point  Targets  (1) 

•  Point  trackers  are  used,  e.g.,  to  track  missile  launches  or  in  infrared 
search  and  track  systems. 

•  Although  different  algorithms  are  used  to  implement  a  tracker,  they  all 
go  through  the  same  basic  steps: 

-  Track  Initiation 

-  Detection-to-Track  Association  (or  Data  Association) 

-  Track  Smoothing 

-  Track  Maintenance. 

•  Track  initiation  is  the  process  of  creating  a  new  track  from  an 
unassociated  detection. 

-  Initially  all  detections  are  used  to  create  new  tracks,  but  once  the  tracker  is 
running,  only  those  that  could  not  be  used  to  update  an  existing  track  are 
used  to  start  new  ones. 

-  A  new  track  is  considered  tentative  until  hits  from  subsequent  updates 
have  been  successfully  associated  with  it. 

-  Once  several  updates  have  been  received,  the  track  is  confirmed. 

•  During  the  last  N  updates,  at  least  M  detections  must  have  been  associated  with 
the  tentative  track  (M=3  and  N=5  are  typical  values). 

-  The  driving  consideration  at  this  stage  is  to  balance  the  probability  of 
detection  and  the  false  alarm  rate. 


Trackers  are  designed  to  follow  the  position  of  a  target  by  responding  to  its  emitted 
or  reflected  radiation.  Most  EO/IR  tracking  systems  contain  the  following  components: 

•  A  sensor  that  collects  radiation  from  the  target  and  generates  a  corresponding 
electrical  signal 

•  Tracker  electronics  that  process  the  sensor  output  and  produce  a  tracking  error 
signal 

•  An  optical  pointing  system  (e.g.,  a  gimbal)  that  allows  the  sensor  to  follow 
target  motion 

•  A  servo  and  stabilization  system  to  control  the  gimbal  position. 

Examples  of  point  target  trackers  include  systems  to  track  missile  launches  and 
infrared  search  and  track  systems.  Many  different  algorithms  are  used  to  implement  a 
point  target  tracker.  Although  different,  they  all  involve  the  same  basic  steps: 

•  Track  Initiation 

•  Data  Association 

•  Track  Smoothing 

•  Track  Maintenance. 

Track  initiation  involves  creating  a  new  track  given  a  new  unassociated  detection. 
Initially  all  detections  are  used  to  create  new  tracks,  but  once  the  tracker  is  running,  only 
those  hits  that  could  not  be  associated  with  an  existing  track  are  used  to  start  new  tracks. 
A  new  track  is  considered  tentative  until  hits  from  subsequent  updates  have  been 
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successfully  associated  with  it.  Tentative  tracks  are  typically  not  shown  to  the  operator  to 
prevent  potential  false  tracks  from  appearing  on  the  screen.  This,  of  course,  causes  some 
delay  in  first  reporting  a  track.  Once  several  updates  have  been  received,  the  track  is 
confirmed.  The  most  common  criterion  for  promoting  a  tentative  track  to  a  confirmed 
track  is  the  so-called  “M-of-N  rule,”  which  states  that  during  the  last  N  updates,  at  least 
M  plots  must  have  been  associated  with  the  tentative  track  (M=3  and  N=5  are  typical 
values).  The  driving  consideration  at  this  stage  is  to  balance  the  probability  of  detection 
and  the  false  alarm  rate. 
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Tracking  Point  Targets  (2) 

•  In  the  data  association  step,  the  tracker  must  determine  which  hits 
should  be  used  to  update  which  tracks. 

-  A  given  hit  may  be  used  to  update  one  track,  or  a  hit  can  be  used  to  tentatively  update 
several  tracks. 

-  The  latter  approach  recognizes  the  uncertainty  surrounding  to  which  track  the  detection 
belongs. 

-  This  produces  multiple  versions  of  each  track  which  are  eventually  “pruned.” 

•  Data  association  can  be  done  in  a  number  of  ways: 

-  By  defining  an  "acceptance  gate"  around  the  current  track  location  and  then  selecting 
either  the  closest  hit  in  the  gate  to  the  predicted  position,  or  the  strongest  hit  in  the  gate. 

-  Statistical  approaches  that  choose  the  most  probable  location  of  a  hit  can  also  be  used. 

•  In  the  track-smoothing  step,  the  latest  track  prediction  is  combined  with 
the  associated  hit  to  provide  a  new  estimate  of  the  target  location. 

-  Algorithms  of  differing  complexity  and  computational  load  can  be  used,  including  alpha- 
beta  trackers,  Kalman  filters,  and  multiple  hypothesis  tracking. 

•  Track  maintenance  involves  deciding  whether  to  terminate  a  track. 

•  Common  approaches  include: 

-  If  the  target  was  not  seen  for  the  past  M  consecutive  update  opportunities 

-  If  the  target  was  not  seen  for  the  past  M  out  of  N  most  recent  update  opportunities 

-  If  the  target's  track  uncertainty  has  grown  beyond  a  certain  threshold. 


In  the  data  association  step,  the  tracker  must  determine  which  hits  should  be  used  to 
update  which  tracks.  This  is  a  trivial  exercise  in  the  case  where  only  a  single  target  is 
being  tracked.  For  those  cases  in  which  multiple  targets  are  being  tracked,  a  given  hit 
may  be  used  to  update  one  track,  or  it  can  be  used  to  tentatively  update  several  tracks, 
recognizing  that  uncertainty  exists  in  knowing  to  which  track  the  detection  actually 
belongs.  This  produces  multiple  versions  of  each  track  that  are  eventually  “pruned.” 
Either  way,  the  first  step  in  the  process  is  to  update  all  the  existing  tracks  by  predicting 
their  new  position  based  on  the  most  recent  state  estimates  (e.g.,  position,  heading,  speed, 
acceleration,  etc.)  and  the  assumed  target  motion  model  (e.g.,  constant  velocity,  constant 
acceleration,  etc.).  Having  updated  the  estimates,  it  is  possible  to  try  to  associate  the  new 
detections  to  the  existing  tracks.  Data  association  can  be  done  by  defining  an  acceptance 
gate  around  the  current  track  location  and  then  selecting  either  the  closest  hit  in  the  gate 
to  the  predicted  position,  or  the  strongest  hit  in  the  gate.  Alternatively,  statistical 
approaches  that  choose  the  most  probable  location  of  a  hit  can  also  be  used. 

Once  a  hit  has  been  associated  with  a  track,  the  next  step  involves  combining  the 
track  prediction  and  the  associated  detection  to  generate  a  new  estimate  of  the  target’s 
location,  as  well  as  to  provide  a  revised  estimate  of  the  errors  in  this  prediction.  A 
variety  of  algorithms  can  be  used  for  this  process,  including: 

•  Alpha-beta  tracker 

•  Kalman  Filter 

•  Multiple  hypothesis  tracker  (MHT) 

•  Interacting  multiple  model  (IMM). 
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The  track  maintenance  step  involves  deciding  whether  to  terminate  a  track.  If  a 
track  was  not  associated  with  a  hit  during  the  data  association  phase,  there  is  a  chance 
that  the  target  may  no  longer  exist.  Alternatively,  however,  the  sensor  may  simply  have 
failed  to  see  the  target  at  that  update,  but  will  find  it  again  on  the  next  one.  Common 
approaches  to  deciding  on  whether  to  terminate  a  track  include: 

•  If  the  target  was  not  seen  for  the  past  M  consecutive  update  opportunities 
(typically  M=3  or  so) 

•  If  the  target  was  not  seen  for  the  past  M  out  of  N  most  recent  update 
opportunities 

•  If  the  target’s  track  uncertainty  has  grown  beyond  a  given  threshold. 
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Tracking  Extended  Targets  (1) 


•  Image-based  trackers  are  used, 
e.g.,  to  track  mobile  objects  (cars) 
on  the  ground  (ISR)  or  for  aim 
point  selection. 

•  The  target  is  initially  located  either 
by  a  human  operator  viewing  a 
display  or  by  some  form  of 
automatic  target  recognition 
system. 

•  After  the  target  is  acquired,  the 
tracking  system  locks  on  to  it  and 
maintains  LOS  autonomously. 

•  A  feedback  control  loop  (the  track 
loop)  continuously  adjusts  the 
gimbal  to  keep  the  target  in  the 
center  of  the  sensor’s  field  of  view. 


-  Target  location  estimation  analyzes  the  sensor’s  imagery  to  determine  the  position  of 
the  target  in  sensor  coordinates. 

-  The  target  location  estimate  is  usually  computed  on  a  small  sub-image  (the  gate)  to 
reduce  the  effects  of  clutter  and  noise  and  to  reduce  the  processing  requirements. 

-  Having  estimated  the  target’s  location,  the  processor  generates  the  gimbal 
commands  to  correct  the  sensor’s  LOS. 

-  The  sensor  generates  a  new  video  stream  and  the  process  repeats. 


Image-based  trackers  are  used  to  track  mobile  objects  on  the  ground  (e.g.,  in  ISR 
missions)  or  for  aim  point  selection.  Like  point  target  trackers,  image -based  tracking 
systems  include  a  sensor,  tracker  electronics,  and  an  optical  pointing  assembly.  With 
image-based  trackers,  a  human  operator  initially  locates  the  target  by  viewing  a  display. 
After  the  target  is  acquired,  a  feedback  control  loop  (the  track  loop)  continuously  adjusts 
the  gimbal  to  keep  the  target  in  the  center  of  the  sensor’s  field  of  view. 

A  generic  tracking  system  is  illustrated  in  this  chart.  Imagery  from  the  sensor  is 
passed  to  the  track  processor.  The  target  location  estimation  process  analyzes  the 
imagery  to  determine  the  position  of  the  target  in  sensor  coordinates.  This  is  done  on  a 
small  sub-image  (the  gate)  to  reduce  the  effects  of  clutter  and  noise  and  to  reduce  the 
processing  requirements.  Having  estimated  the  target’s  location,  the  processor  generates 
the  gimbal  commands  to  correct  the  sensor’s  line  of  sight  (LOS).  The  sensor  then 
generates  a  new  video  stream  and  the  process  repeats. 
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Tracking  Extended  Targets  (2) 

•  No  universal  tracking  system  exists  that  can  meet  performance  requirements 
specified  for  all  applications. 

•  Common  Tracker  types  (target  location  estimation)  include: 

-  Edge  Detection  Trackers:  Identifies  points  in  the  image  where  the  brightness  changes 
abruptly  thereby  defining  edges  of  the  target 

-  Centroid  Trackers:  Determines  target  aim  point  by  computing  the  geometric  centroid  of  the 
target 

-  Area  Correlation  Trackers:  Compare  patterns  via  an  autocorrelation  function  to  develop  an 
error  signal 

-  Moving  Target  Indicators  (MTI):  Subtracts  the  present  video  frame  from  the  preceding  one 
to  isolate  moving  targets 

-  Multi-mode  Trackers:  Combine  point,  area,  and  MTI  schemes  together  as  necessary 

-  Feature  Based  Trackers:  the  most  current  trend  in  advanced  tracker  development. 

•  General  Lock-on  Requirements  include 

-  Signal-to-Noise  Ratio  1  Determine  fundamental  performance  limits 

-  Signal-to-Clutter  Ratio  J 

r  Set  by  designer  to  achieve  more  stable  tracking  in  clutter 

-  Pixels  on  Target  J 

•  Kalman  filtering  can  provide  optimum  position  estimate  in  noisy  environments. 

-  Often  used  in  trackers  required  to  follow  maneuvering  airborne  targets  from  high 
performance  aircraft  platforms. 


The  heart  of  the  tracker  is  in  the  feedback  control  loop  or  the  track  loop.  Common 
image -based  tracker  algorithms  include:  edge  detection,  centroid  tracking,  area 
correlation  tracking,  moving  target  indicators  (MTIs),  multi-mode  trackers,  and  feature- 
based  algorithms. 


Tracking  Summary 

•  Examples  of  point  target  trackers  include  systems  to  track  missile  launches  and 
infrared  search  and  track  systems. 

•  Many  different  algorithms  are  used  to  implement  point  trackers.  Although 
different,  they  all  involve  the  same  basic  steps:  track  initiation,  data  association, 
track  smoothing  and  track  maintenance. 

•  Image-based  trackers  are  used  to  track  mobile  objects  on  the  ground  (ISR 
missions)  or  for  aim  point  selection. 

•  Usually  with  image-based  trackers,  a  human  operator  initially  locates  the  target 
by  viewing  a  display. 

•  After  the  target  is  acquired,  a  feedback  control  loop  (the  track  loop) 
continuously  adjusts  the  gimbal  to  keep  the  target  in  the  center  of  the  sensor’s 
field  of  view. 

•  Common  image-based  tracker  algorithms  include:  edge  detection,  centroid 
tracking,  area  correlation  tracking,  MTIs,  multi-mode  trackers,  and  feature- 
based  algorithms. 
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E.  Latest  Trends  in  Sensors 


Focal  Plane  Arrays 


Trends: 


•  Large  Area  and  High  Resolution 

-  Number  of  pixels  vs.  Pixel  Size 

-  Argus/Gorgon  Stare/Angel  Fire 

•  Uncooled  IRFPAs 

-  Microbolometers 

•  Extended  Wavelength  Response 
(VLWIR) 

•  lll-V  Based  FPAs 

-  QWIPs,  T2SLSs,  QDIPs 


Advanced  Processing 

-  Super-Resolution  (microscanning) 
On-Chip  Processing 

-  Non-uniformity  Correction  (NUC) 

-  Analog  to  Digital  Conversion  (ADC) 
Adaptive  multi/hyperspectral  imaging 

-  Independently  tune  spectral  response  of 
each  pixel 

-  Array  of  electronically  programmable 
microspectrometers 


Driven  by  missions  such  as  Persistent  Surveillance,  an  important  recent  trend  in 
EO/IR  sensors  is  the  development  of  large-area,  high-resolution  cameras.  For  sensors 
operating  in  the  visible  band,  this  is  simpler  than  for  those  operating  in  the  mid¬ 
wavelength  infrared  (MWIR)  or  long-wavelength  infrared  (LWIR)  bands.  In  the  visible 
band,  silicon  can  be  used  for  both  the  sensing  material  and  for  readout  electronics.  This 
eliminates  issues  such  as  the  potential  thermal  mismatch  between  the  two.  For  EO 
systems  it  is  also  possible  to  group  individual  cameras  together  and  then  combine  the 
separate  images  into  one.  The  current  state  of  the  art  in  MWIR  focal  plane  arrays  (FPAs) 
is  roughly  4,000  by  4,000  detectors;  for  FWIR  FPAs,  it  is  closer  to  2,000  by  2,000.  The 
ARGUS  sensor  being  developed  by  DARPA  has  over  2  gigapixels. 

To  reduce  the  amount  of  thermally  generated  noise  in  FWIR  FPAs,  these  cameras 
operate  at  low  temperatures.  The  required  cryogenic  equipment  has  important  Size, 
Weight,  and  Power  (SWaP)  implications  at  the  system  level  and  can  affect  system 
reliability.  For  some  applications,  a  small  lightweight  camera  that  does  not  consume 
much  power  is  required.  To  address  this  need,  considerable  effort  has  gone  into  the 
development  of  uncooled  sensors.  A  recent  NVESD  program  to  develop  uncooled 
microbolometers  with  detectors  as  small  as  17  pm  and  array  formats  as  large  as  1024  x 
768  is  addressing  the  tradeoff  between  device  sensitivity  and  temporal  response.  The 
goals  of  the  program  are  a  sensitivity  of  35  mK  and  a  time  constant  of  10  ms. 

The  detector  material  of  choice  for  FWIR  applications  is  mercury  cadmium  telluride 
(HgCdTe).  The  spectral  band  of  these  devices  is  determined  by  adjusting  the 
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composition  of  the  material,  which  can  be  difficult  for  longer  cut-off  wavelengths.  An 
alternative  approach  involves  the  use  of  so-called  superlattices  of  III-V  semiconductors. 
This  device  structure  consists  of  alternating  layers  of,  e.g.,  gallium  arsenide  (GaAs)  and 
aluminum  gallium  arsenide  (AlGaAs).  For  these  compounds,  the  spectral  cut-off 
wavelength  is  changed  by  altering  the  layer  thickness,  which  can  be  done  very  precisely. 

In  the  past,  sensor  performance  was  limited  by  detector  quality,  due  to  detector 
material  and  fabrication.  The  improvement  in  these  areas  has  reached  the  point  where 
future  gains  in  system  performance  will  be  achieved  through  advanced  processing 
techniques.  One  such  technique  is  known  as  super-resolution.  The  objective  of  super¬ 
resolution  is  to  increase  the  spatial  resolution  of  the  sensor  without  physically  altering  the 
optics  or  focal  plane.  The  technique  involves  the  use  of  successive  frames  from  a 
spatially  under-sampled  imager  to  process  and  construct  a  higher-resolution  image.  A 
number  of  fielded  systems  incorporate  this  concept  to  improve  range  performance.  An 
example  is  the  Lockheed  Martin’s  Hawkeye  system.  This  system  is  gimbaled  and 
includes  an  indium  antimonide  (InSb)  MWIR  FLIR.  The  super-resolution  algorithms 
result  in  range  performance  improvements  of  the  order  of  20  percent  to  40  percent, 
depending  on  which  optical  field  of  view  (FOV)  the  system  is  using.  The  Northrop 
Grumman  LITENING  AT  pod  also  utilizes  similar  algorithms  in  an  application  for 
tactical  jet  aircraft  targeting  pods. 
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Army  Third  Generation  FLIR 


Third-Generation  imager  specifications  •  Benefits  of  Gen  3  FLIR 

Better  day  signatures  (MWIR  band) 


Array  format 

1280x720 

Better  transmission  (always  have  the  best  band) 

Full  Frame  Rate 

60  Hz 

Better  countermeasure  rejection  (against  single  band 

Window  frame  rate  (64  x  64) 

480  Hz 

lasers) 

F  /# 

F/2.5  and  F/6.0 

Longer  range  identification  (NFOV  MWIR  Channel) 

Pixel  pitch 

20  pm 

Faster  Search  (WFOV  LWIR  has  shorter  integration  time) 

Spectral  Band-pass 

LW: 

7.8-10.5  pm 

Less  Motion  Smearing  (LWIR  has  shorter  integration  time) 

MW; 

3.4-5.0  pm 

Wider  Dynamic  Range  (LWIR  channel  has  more  sensitivity) 

NEAT  @  F/6.0.  300K 

LW; 

35  mK 

Opportunity  for  Multi-color  fusion 

MW; 

35  mK  * 

Disadvantages  of  Gen  3  FLIRs 

NEAT  @  F/2.5.  300K 

LW; 

25  mK 

Increased  Cost  and  Complexity 

MW: 

25  mK 

•  Two-color  FPAs 

Dynamic  Range 

>78  dB 

•  Dual  cold  shield  design 

Scene  temperatures 

227K  to  360K 

•  Dual  channel  optics 

Operating  temperature 

>80  K 

•  More  signal  processing 

An  example  of  a  proposed  system  that  incorporates  many  of  the  developments 
described  on  the  preceding  chart  is  the  Army’s  third-generation  FLIR.  The  FPA  is  a  large 
format  (1280  x  720)  staring  array  with  a  detector  pitch  of  20  pm.  In  addition,  the  device 
architecture,  which  is  shown  above,  allows  operation  in  both  the  MWIR  and  the  LWIR 
spectral  bands.  The  system  has  a  dual  F/#  optical  system  (F/2.5  and  F/6.0).  The  key 
advantage  of  two-color  imaging  infrared  focal  plane  arrays  (IRFPAs)  is  the  ability  to 
more  easily  detect  and  identify  obscured  targets  in  cluttered  backgrounds.  Other 
advantages  are  listed  above.  These  come  at  the  expense  of  increased  cost  and 
complexity. 
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SWIR  Imagers  (1) 
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•  Radiance  of  the  night  sky 

-  The  origins  and  expected  intensities  of 
hydroxyl  (OH)  are  shown  by  the  vertical 
lines 


*  Military  Applications  of  SWIR  Imaging 

-  Low  Light  /  Night  Vision  Imaging 

-  Detection  of  Laser  Designators 

-  Covert  Laser  Illumination 

-  Laser  Gated  Imaging 

-  Threat  Warning  Detection 

-  ISR 

-  Covert  Tagging 

*  Size,  Weight  and  Power 


As  discussed  on  page  7,  the  infrared  portion  of  the  electromagnetic  spectrum  is 
generally  subdivided  into  numerous  bands,  including  the  near-infrared  (NIR),  the  short- 
wavelength  infrared  (SWIR),  the  mid-wavelength  infrared  (MWIR),  the  long-wavelength 
infrared  (LWIR),  and  the  very  long-wavelength  infrared  (VLWIR).  The  SWIR  band 
extends  from  1.0  pm  to  3.0  pm  and  is  defined  by  the  carbon  dioxide  (CO2)  and  water 
(H20)  absorption  lines  at  both  the  short-  and  long- wavelength  ends  of  the  band. 

Radiation  in  the  SWIR  band  is  often  referred  to  as  “airglow”  or  “nightglow”  and  is 
non-thermal  radiation  emitted  by  the  Earth’s  atmosphere.  Its  spectral  radiance  is  of  the 
order  of  1  pW/(cm2*Sr*pm)  with  a  spectral  distribution  characteristic  of  molecular 
emission.  The  primary  source  of  this  emission  was  identified  by  A.B.  Meinel  in  1950  as 
rotation-vibration  bands  of  the  hydroxyl  (OH)  molecule. 

Military  applications  of  SWIR  imaging  include  low-light/night-vision  imaging, 
detection  of  laser  designators,  covert  laser  illumination,  laser-gated  imaging,  threat- 
warning  detection,  ISR,  and  covert  tagging. 
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SWIR  Imagers  (2) 


An  image  of  a  hotel  in  haze  (range  =  2.4  km) 
Scattering  due  to  haze  primarily  affects 
visible  systems,  scattering  due  to  fog  and 
dust  can  affect  both  visible  and  IR 
radiation. 

Which  band  performs  better  in  a  given 
situation  depends  on  the  details  of  the 
aerosol  present. 


SWIR  sensors  rely  primarily  on  reflected 
radiation  similar  to  visible  cameras. 
SWIR  cameras  produce  finer  resolution 
than  imagers  operating  in  the  LWIR. 

This  can  lead  to  improved  target  ID. 
SWIR  imagers  can  be  combined  with 
illuminators  for  imaging  inside  buildings 
and  caves. 


Most  glass  is  opaque  for  X  >  3.5  pm. 

Glass  windows  are  transparent  in  the  SWIR 
band  but  not  in  the  LWIR. 

Some  transparency  in  the  MWIR  but  poor 
optical  transmission  degrades  the  image. 
This  is  significant  operationally  but  also 
has  significant  system  implications. 


Image  Sources:  Goodrich  Corp.,  and  Sensors  Unlimited,  Inc.,  "The  Advantages  of  SWIR 
Imaging,"  http://www.sensorsinc.com/images.html;  Photonics  Spectra  magazine 


The  frequency  range  where  light  scattering  is  most  prominent  depends  upon  the  size 
of  the  particles/aerosols  doing  the  scattering.  Haze  refers  to  obscuration  of  the  lower 
atmosphere  due  to  particles/aerosols  in  the  0.01  -  0.1  pm  size  range.  For  fog  and  dust, 
the  particles/aerosols  are  larger;  typically,  0.5  -  100  pm  for  fog  and  1-100  pm  for  dust. 

The  ability  of  a  SWIR  camera  to  image  through  haze  is  illustrated  in  the  top  two 
figures,  which  show  both  a  visible  and  a  SWIR  image  of  a  hotel  in  haze  at  a  range  of 
roughly  2.4  km.  The  image  was  taken  in  late  afternoon  before  dusk  about  150  feet  above 
the  ground.  As  can  be  seen,  the  hotel  is  barely  discernible  in  the  visible,  but  is  relatively 
clear  in  the  SWIR  band.  The  reason  for  this  difference  is  that,  in  hazy  conditions,  the 
scattering  efficiency  for  visible  light  is  much  higher  than  it  is  in  the  SWIR  band; 
consequently,  the  longer  wavelengths  of  the  SWIR  band  transmit  more  readily. 

Sensors  operating  in  the  SWIR  spectral  region  detect  differences  in  ambient  light 
reflected  from  the  target  and  background,  similar  to  a  camera  operating  in  the  visible. 
Thus  a  SWIR  image  looks  much  more  like  a  visible  image  than  does  one  produced  by  a 
thermal  imager.  This  is  illustrated  in  the  middle  pair  of  figures  above.  These  figures  also 
show  the  increased  resolution  possible  with  a  SWIR  camera  compared  to  that  with  a 
long-wavelength  infrared  imager.  This  increased  resolution,  in  turn,  leads  to  improved 
target  identification. 

Optical  glasses  are  transparent  to  light  with  wavelengths  from  around  0.2  to  3.5  pm. 
The  optical  transmission  then  goes  to  zero  and  stays  at  zero  through  the  LWIR  and  very 
long  wavelength  infrared  (VLWIR)  spectral  regions.  This  poor  transmission  is  a  result  of 
strong  absorption  due  to  the  presence  of  water  and  hydroxyl  ions.  Thus  glass  windows 
are  not  transparent  in  the  LWIR  but  are  in  the  SWIR  spectral  band.  With  some  MWIR 
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cameras,  it  is  possible  to  see  through  glass,  but  the  poor  optical  transmission  significantly 
degrades  the  image.  The  difference  between  the  LWIR  and  SWIR  spectral  bands  in  terms 
of  their  ability  to  image  through  glass  is  illustrated  in  the  bottom  pair  of  figures.  This  is 
important  not  only  from  an  operational  point-of-view  where  it  is  often  necessary  to  image 
into  and  out  of  windows,  windshields,  etc.,  but  it  also  has  significant  system  implications. 
The  fact  that  standard  glass  optics  and  lenses  can  be  used  in  the  SWIR  band  is  an 
important  advantage  in  terms  of  cost  and  complexity  compared  to  both  MWIR  and 
LWIR,  which  require  optical  systems  that  use  more  exotic  materials  such  as  germanium 
(Ge)  and  sapphire  (AI2O3). 
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Multi-Spectral  and  Hyper-Spectral  Sensors  (1) 


•  Multi-spectral  and  hyper-spectral  sensors 
offer  more  than  red,  green,  and  blue. 

-  Multi-spectral  typically  means  up  to  8-12  bands, 

-  Hyper-spectral  >  50  bands, 

•  For  multi-spectral  systems,  the  bands  are 
chosen  with  very  specific  applications  in 
mind. 

-  Chlorophyll,  algae,  sediment 

-  Oil  spills,  fire  detection 

•  For  hyper-spectral  systems,  custom  band 
choices  may  be  offered. 

•  Multi-spectral  and  hyper-spectral  sensors 
typically  scan  across  the  direction  of 
flight. 

•  If  the  number  of  bands  is  small,  5-8, 
pushbroom  scanners  can  be  used. 


Dalsa  5-band  chip 
(up  to  12,000+ 
detectors  wide) 


Argon  Enhanced  Airborne  Multispectral  Scanner 
(0.6  mr  IFOV) 


Argon  Airborne  Hyperspectral  Scanner 
(1.25  or  2.5  mr  IFOV) 


Multi-spectral  sensors  produce  images  with  a  few  discrete  and  somewhat  broad 
spectral  bands,  e.g.,  Vis,  NIR,  MWIR,  and/or  LWIR.  Hyper-spectral  sensors,  on  the 
other  hand,  collect  image  data  simultaneously  in  dozens  or  hundreds  of  narrow,  adjacent 
spectral  bands.  These  measurements  make  it  possible  to  derive  a  continuous  spectrum  for 
each  image  cell.  So  a  sensor  with  only  20  bands  could  also  be  hyper-spectral  if  it  covered 
the  range  from  500  to  700  nm  with  20  bands  each  10  nm  wide.  A  sensor  with  20  discrete 
bands  covering  different  parts  of  the  Vis,  NIR,  SWIR,  MWIR,  and  LWIR  spectra  would 
be  considered  multi-spectral. 

Hyper-spectral  sensors  collect  information  as  a  set  of  images,  one  for  each  spectral 
band.  These  spectral  images  are  then  combined  and  form  a  3D  hyper-spectral  data  cube 
for  processing  and  analysis  -  in  other  words,  the  spatial  data  are  in  the  XY  plane;  the 
spectral  data,  the  Z-axis.  These  systems  typically  have  only  moderate  spatial  resolution 
in  each  of  up  to  200  spectral  sub-bands,  with  each  spectral  sub-band  having  a  width  of 
about  10  nm. 


Hyper-spectral  images  are  produced  by  instruments  called  imaging  spectrometers. 
The  most  common  approach  to  creating  a  hyper-spectral  data  cube  takes  advantage  of  2D 
FPAs.  The  camera  first  maps  an  image  strip  onto  a  row  of  pixels  on  the  FPA.  Each  pixel 
in  this  row  is  simultaneously  spread  into  a  Z  column  of  spectral  data  and  then  that 
(spectral)  frame  is  read.  Then  the  imager  scans  to  the  next  row.  This  is  repeated  until  a 
2D  spatial  image  (in  the  X-Y  plane)  is  built  up.  Thus,  for  example,  a  320  x  240  array 
could  capture  320  spatial  pixels  with  240  spectral  sub-bands  associated  with  each  pixel. 
The  point  of  obtaining  an  image  cube  is  to  allow  one  to  compare  the  spectral  intensity 
plot  obtained  from  each  pixel  with  a  stored  library  of  spectral  reflectivity  data. 
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Multi -Spectral  and  Hyper-Spectral  Sensors  (2) 


•  VIMS  is  located  on  the  Cassini  Orbiter  to  provide  2D 
hyperspectral  high  resolution  images  to  study  Saturn’s  rings. 

•  VIMS  makes  use  of  a  reflective  telescope  and  pushbroom 
imaging  with  a  pointable  primary  mirror. 

•  The  spectral  range  is  from  325  to  1025  nm,  with  spectral 
resolution  of  1.46  nm. 


•  The  objective  is  to  allow  one  to  compare  the 
spectral  intensity  plot  obtained  from  each 
pixel  with  a  stored  library  of  spectral 
reflectivity  data. 

•  Challenges  remain: 

-  Atmospheric  transmission  correction 

•  Altitude  Dependency 

•  Supporting  Meteorological  Instrumentation 

•  Measurement  Latency 

-  Database  development 

•  Material  Spectra  Libraries 

•  Weathering  Effects  on  Materials 

•  Possible  CMs  (design  camouflage  materials 
that  match  the  background) 


Military  systems  are  largely  focused  on  the  SWIR  spectral  band  for  two  reasons. 
First,  most  materials  have  high  reflectivities  in  this  region,  and  their  reflectivity  spectra 
differ  strongly  due  to  material-dependant  molecular-absorption  bands.  Second,  the  strong 
solar  illumination  available  during  the  day  provides  the  very  high  signal  levels  needed  to 
achieve  an  adequate  SNR  given  the  lack  of  light  when  working  in  very  narrow  spectral 
bands. 

Numerous  hyper- spectral  data  collection  systems  have  been  developed  -  most 
designed  for  airborne  usage,  but  some  are  satellite  mounted.  Much  of  the  development 
work  in  this  area  is  in  the  field  of  remote  sensing.  A  typical  system  might  have  200 
spectral  bands  each  with  10-nm  spectral  resolution.  In  addition,  the  system  might  may 
have  1  mrad  spatial  resolution  and  provide  a  50  to  70  degree  sweep.  For  systems 
operating  in  the  SWIR  band,  the  FPA  is  typically  InGaAs  or  HgCdTe  while  those 
operating  in  the  MWIR  band  use  HgCdTe  or  InSb  FPAs. 

The  primary  advantage  to  hyper-spectral  imaging  is  that,  because  an  entire  spectrum 
is  acquired  at  each  point,  the  operator  needs  no  prior  knowledge  of  the  sample,  and  post¬ 
processing  allows  all  available  information  from  the  dataset  to  be  mined.  Hyper-spectral 
imaging  can  also  take  advantage  of  the  spatial  relationships  among  the  different  spectra  in 
a  neighborhood,  allowing  more  elaborate  spectral-spatial  models  for  a  more  accurate 
segmentation  and  classification  of  the  image. 

The  primary  disadvantages  are  cost  and  complexity. 
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Latest  Trends  Summary 

•  Missions  such  as  persistent  surveillance  are  driving  the  need  for  high  resolution 
sensors  with  extremely  large  FOVs. 

•  Uncooled  microbolometers  with  detectors  as  small  as  17  pm  and  array  formats 
as  large  as  1024  x  768  will  be  important  for  applications  with  difficult  SWaP 
constraints. 

•  The  use  of  so-called  III-V  superlattices,  such  as  AlGaAs/GaAs,  allow  one  to 
change  the  spectral  response  of  the  device  by  changing  the  layer  thickness. 

•  Because  detector  materials  are  operating  close  to  theoretical  limits,  future  gains 
in  system  performance  will  be  achieved  through  advanced  processing,  such  as 
super-resolution. 

•  The  SWIR  band  extends  from  1.0  pm  to  3.0  pm  and  is  defined  by  the  CO2  and 
H20  absorption  lines  at  both  the  short-  and  the  long-wavelength  ends  of  the 
band. 

•  Radiation  in  the  SWIR  band  is  often  referred  to  as  “airglow”  or  “nightglow”  and 
is  non-thermal  radiation  emitted  by  the  Earth’s  atmosphere. 

•  Military  applications  of  SWIR  imaging  include  low-light/night-vision  imaging, 
detection  of  laser  designators,  covert  laser  illumination,  laser-gated  imaging, 
threat-warning  detection,  ISR,  and  covert  tagging. 

•  Hyper-spectral  imaging  allows  one  to  simultaneously  gather  both  spatial  and 
spectroscopic  information  for  more  accurate  segmentation  and  classification  of 
an  image. 
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F.  Sampling,  Aliasing,  and  Reconstruction 


Background 

•  Most  modern  sensors  are  “sampled”  producing  an  array  of  numbers  that 
represent  an  image  (like  the  CD  representation  of  music). 

•  This  is  in  contrast  to  older  technologies,  such  as  film  or  image  intensifiers  (both 
still  used),  which  represent  the  image  continuously  (like  a  vinyl  LP  or  audio  tape). 

•  Because  the  (at  least  conceptually)  continuous  image  is  sampled,  the  data 
depend  on  the  placement  of  the  samples,  or  the  “phase”  of  the  samples  as 
shown  in  the  figures  below:  the  blue  and  red  circles  denote  different  possible 
samplings  of  the  image. 

•  The  image  artifacts  produced  by  the  sampling  are  collectively  referred  to  as 
“aliasing.” 


GSD,  the  distance  on  the  ground  between  samples,  is  often  used  as  shorthand  for  the 
resolution  and  hence  performance  of  a  digital  system.  Although  the  resolution  is  usually 
not  far  from  the  GSD,  this  over-simplifies  the  problem.  For  example,  if  the  system  is 
unaliased  (which  is  possible  for  sufficiently  high  sampling  or  sufficiently  poor  optics), 
the  resolution  will  be  poorer  than  the  GSD.  How  much  poorer  will  be  determined  by  the 
optics  and  detector  MTF.  If  the  system  is  aliased,  the  resolution  will  be  determined  by 
how  the  aliasing  affects  task  performance;  it  may  be  greater  or  lesser  than  indicated  by 
the  GSD. 

It  is  commonplace  to  assert  that  spatial  frequencies  below  the  Nyquist  limit  will  be 
accurately  reproduced;  although  true  for  non-aliased  systems,  this  depends  on  the  task  for 
aliased  systems.  In  general,  one  may  expect  the  probability  of  task  performance  will 
degrade  as  the  Nyquist  frequency  is  approached. 

Similarly,  the  task  may  be  able  to  be  performed  above  the  Nyquist  frequency,  but,  in 
many  cases,  the  performance  continues  to  degrade  as  the  frequency  increases  above 
Nyquist.  Often,  task  performance  is  statistical:  for  some  phases  of  the  sampling,  the  task 
is  performable;  for  others,  not. 

Finally,  it  is  insufficient  to  compare  the  “characteristic  dimensions”  of  the  object  of 
interest  with  the  GSD.  Real  objects  represent  a  range  of  spatial  frequencies  ranging 
above  and  below  characteristic  dimensions.  A  complete  analysis  must  take  this  into 
account  or  be  made  in  the  spatial,  not  frequency,  domain. 
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How  Does  Aliasing  Look? 


•  The  Moire  pattern.  The  strong  high  spatial 
frequency  information  represented  by  the 
brick  pattern  produces  a  low-frequency 
oscillating  pattern  and  distortions  of  the 
high-frequency  pattern  (brick  shape). 

•  Interruptions  of  linear  features:  the  tele¬ 
phone  lines  are  intermittently  revealed. 


Unaliased  versus  aliased: 


•  Confusions  of  detail.  The  bar  target  con 
sists  of  groups  of  3-bars.  Sampling  ef¬ 
fects  can  wipe  out  the  pattern  or  even 
make  it  appear  as  a  2-bar  pattern. 


HHiN« 

wm*** 


The  images  above8  illustrate  what  aliasing  looks  like  in  an  image.  The  top  left 
image  shows  an  accurate,  high-sample  rate,  depicting  a  very  regularly  patterned  image. 
Each  brick  is  clearly  delineated  and  each  brick  is  of  the  correct  size.  The  top  right  image 
is  down-sampled  version  of  the  one  on  the  left.  When  sampling  the  brick,  a  sample  may 
not  land  on  the  white  mortar  between  bricks  that  define  the  brick’s  outline.  The  net  result 
of  such  coarse  sampling  is  the  introduction  of  waves  of  alternating  dark  and  light,  termed 
a  “Moire  pattern.”  The  precise  location  of  the  bands  depends  on  exactly  how  the  samples 
fall;  this  is  termed  “phasing”  of  the  samples. 

A  close  examination  of  the  image  will  also  show  apparent  distortions  of  the  apparent 
shapes  of  the  bricks,  which  also  varies  depending  on  the  position  of  the  brick  on  the  wall, 
which  determines  the  phasing  of  the  samples. 

If  the  task  is  to  determine  the  size  and  shape  of  the  bricks  or  to  confirm  that  each 
brick  is  the  same  size,  then  the  task  is  severely  affected  by  aliasing.  If  the  task  is  to 
determine  whether  the  wall  is  brick  or  cinder  block,  the  task  is  not  affected  by  aliasing  at 
all. 


Source:  Widipedia. 
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Where  does  Aliasing  come  from? 

•  In  the  figure 

-  The  red  samples#  see  a  solid  blue  expanse. 

-  The  yellow  samples  see  a  solid  orange 
expanse. 

-  The  answer  depends  on  the  phase  of  the 
samples: 

if  the  samples  were  shifted,  we  might  get 
brown  with  either  the  red  or  yellow  samples. 

•  At  least  in  the  first  case,  combining  both  red  and  yellow 
samples  would  provide  a  correct  interpretation  of  the  image 
(blue-orange-blue) 

-  We  can  (usually)  represent  correctly  anything  with  2  samples  per 
“cycle.” 

-  This  is  the  folk  version  of  the  Nyquist  theorem. 

•  Some  of  the  effects  can  only  be  explicated  by  looking  at  the 
math. 


As  noted  several  times  in  the  preceding  charts,  aliasing  is  the  result  of  sampling. 
The  figure  shows  a  blue-and-orange  striped  “target.”  If  only  the  samples  shown  as  red 
circles  in  the  upper  figure  were  included  in  the  sampled  data  set,  the  image  would  be 
interpreted  as  being  entirely  blue;  if  only  the  samples  shown  as  yellow  circles  were  taken, 
the  target  would  be  entirely  orange.  The  red  and  yellow  samples  differ  only  in  their 
phase  with  respect  to  the  pattern  of  the  target. 

For  the  phases  in  the  top  image,  if  one  had  both  red  samples  and  yellow  samples, 
one  would  have  correctly  deduced  that  the  target  had  blue  and  orange  panels.  This 
corresponds,  in  a  quick-and-dirty  version  of  the  Nyquist  sampling  theorem,  to  having  (at 
least)  one  sample  per  characteristic  feature. 

One  needs,  in  general,  to  have  more  than  one  sample  per  characteristic  feature.  This 
is  illustrated  by  the  lower  figure,  which  has  the  same  number  of  samples  (red  and  yellow) 
as  the  upper  figure  but  for  which  the  samples  are  shifted  -  they  have  a  different  phase.  In 
this  case,  each  sample  overlaps  an  equal  amount  of  blue  and  orange;  the  structure  is  again 
lost  and  the  color  distorted. 
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Sampling  and  Reconstruction 

•  All  digital  systems  are  sampled  systems. 

-  For  a  given  continuous  signal  g(x),  there  is  a  set  of  sample  locations  {xm} 
and  sample  values  gm  =  g(xj. 

-  The  sample  locations  are  typically  (but  not  always)  chosen  to  have  a 
uniform  spacing  between  samples. 

xm  =  x0+  m*S  where  S  =  sample  separation  and  x0  the  initial  phase 

-  The  sampled  signal  could  be  represented  as  a  series  of  delta  functions  or 
we  can  “reconstruct”  the  signal  with  smoother  functions. 

gMU(x)  =  Y,gmSS(x-xm) 

&  reconstruction  (x)  =  ^gmh  (x-xj 
m  m 

•  In  the  Fourier  (frequency)  domain,  if  the  Fourier  transform  of  the 
original  signal  g(x)  is  G(f)  then  (letting  x0  =  0  for  convenience) 

<W/)=  ZG(/-«/s)  GrecomMton{f)  =  H{f)fjG{f-nfs) 

-  Where  H(f)  is  the  Fourier  transform  of  the  smooth  reconstruction  function. 

-  The  sampled  signal  consists  of  the  original  plus  an  infinite  series  of 
sideband  copies  of  the  original. 

-  The  reconstruction  controls  how  one  deals  with  the  sidebands  and  is  an 
important  part  of  the  evaluation  of  any  sampled  system. 


Digital  systems  are  a  subset  of  sampled  systems.  However,  the  digitization 
(quantization)  is  separable  from  sampling  itself.  Similarly,  other  techniques,  such  as  fast 
Fourier  transforms  (FFT),  although  extremely  useful  for  real-time  processing,  are 
irrelevant  for  the  primary  issues  addressed  in  this  series  of  presentations. 

Digital  signal  processing  emphasizes  the  direct  manipulation  of  the  digitized 
samples,  transforming,  for  example,  a  N  x  N  array  of  samples  representing  an  image  into 
another  N  x  N  array  of  modified  samples.  The  emphasis  here,  in  contrast,  is  on  the 
continuous  perspective. 

The  samples  are  taken  from  a  continuous  function;  the  goal  is  to  extract  the  most 
information  possible  from  those  samples  (without  introducing  artifacts)  about  the 
continuous  image.  In  addition,  having,  at  least  conceptually,  reconstructed  the  original 
image,  the  image  can  be  manipulated  in  a  variety  of  ways  using  the  wider  range  tools 
available  for  a  continuous  image.  These  manipulations  include,  but  are  not  limited  to, 
smoothly  interpolating  the  image  so  that  it  can  be  added  to  other  images  or  corrected  for 
various  image  distortions. 

These  manipulations  may  actually  be  done  (indeed,  must  be  done  in  practice)  by 
digitally  manipulating  another  array  of  data  that  is  larger  than  the  original  array  (MN  x 
MN).  However,  the  manipulations  can  be  represented  theoretically  by  smooth 
transformations,  and  the  re-digitization  needed  to  perform  the  calculations  is  a  logically 
separate  part. 
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Unaliased  and  Aliased  Systems 

•  In  the  most  desirable  case,  the  original  signal  is  separated  from  the  copies. 
This  is  the  unaliased  case.  The  original  information  is  present  and,  in 
principle,  recoverable. 

-  This  requires  that  the  original  signal,  G(f),  is  band-limited  with  a  maximum 
frequency,  fmax  :  and  that  the  sample  rate  is  sufficiently  high. 

-  The  sampling  rate  must  be  twice  the  maximum  frequency  to  keep  the  copies 
from  overlapping. 

-  The  Nyquist  frequency  is  defined  as  !4  the  sampling  rate,  /max  <  //  ^  =  ftfyqwst 

•  When  these  conditions  do  not  apply,  the  copies  will  overlap:  the  region  of 
overlap  is  the  aliased  part  of  the  frequency  domain.  /max  >  fNyquist 

-  Moire  patterns  and  other  sampling  anomalies  are  now  inevitable. 

Not  a  true  copy  in  central  region 
when  there  is  aliasing 


True  copy  in  central  region 
if  there  is  no  aliasing 


No  signal  with  finite  support  (so  no  real  image  in  a  camera)  is  literally  band-limited. 
However,  within  the  approximation  that  there  is  an  effective  maximum  frequency,  fmax,  a 
particular  input  frequency,  fnput,  is  unaffected  by  the  aliased  sidebands  if fmax  <  fs  /2.  If 
fmax  <  fs,  some  region  of  frequencies  is  unaliased.  This  is  important  for  some  applications. 
For  example,  in  point  target  detection  (such  as  missile  launches),  the  source  image  on  the 
focal  plane  is  determined  by  the  optical  blur  of  the  system  so  that,  if  the  system  is  not  too 
aliased  (meaning  that  the  blur  circle  is  not  too  small),  the  blur  will  cover  more  than  one 
detector.  It  is  therefore  possible  to  estimate  (if  the  background  clutter  and  noise  are  not 
too  high)  the  target’s  location  by  a  suitable  average  of  the  different  sample  values. 
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Ambiguity  in  Aliased  Systems 

•  If  the  system  is  aliased,  with  fmax  >  fNyquist=  1/2S,  the  sampling  itself  is 
ambiguous:  the  samples  cannot  distinguish  between  frequencies  above 
Nyquist  and  frequencies  below  Nyquist 

-  Samples  at  x=  m*S 

-  Let  f+/_  =  fNyquist  +/-  df 

F°r  g(x )  =  A  cos( 2nf+x)  +  B cos(2xf_  x) 

the  sample  values,  gm,  are  given  by: 

gm=(-l)m(A  +  B)cos(27rniSSf) 

the  samples  cannot  distinguish  between  the  higher  and  lower  frequency. 

•  The  delta-function  reconstruction  gives  equal  weight  to  f+  and  f 

•  Non-trivial  reconstructions  address  the  ambiguity  by  trying  to  assign 
relative  weights  to  the  higher  (f  >  fNyq uist)  and  lower  frequencies 

if  ^  f Nyquist )■ 

-  Most  reconstructions  favor  the  lower  frequencies. 

-  In  some  cases,  the  higher  frequencies  might  be  the  real  ones! 

•  There  is  no  absolute  theoretical  approach  to  take:  it  will  depend  on  the 
task. 


It  is  commonly  stated  that  frequencies  higher  than  Nyquist  “cannot”  be  represented 
and  that  frequencies  below  Nyquist  can  be  reliably  represented.  Both  statements  are  over 
simplifications.  Even  in  the  unaliased  case,  a  correct  statement  of  the  sampling  theorem 
is  that,  if  the  original  signal  lies  entirely  in  one  interval  [M  fNyquist,  (M+l)  fNyquist  ],  then  it 
can  be  perfectly  reconstructed. 

If  the  signal  was  known  to  consist  entirely  of  frequencies  above  Nyquist,  then  the 
frequencies  below  Nyquist  should  be  eliminated. 

The  problem  is  that  frequencies  above  and  below  Nyquist  produce  the  same  samples 
for  cosines  and  samples  differing  only  by  an  overall  phase  for  sine  terms. 
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Aliasing  in  Imaging  Systems 

•  The  challenge  in  understanding  aliasing  in  imaging  systems  is  the  conflict 
between  two  extremes  (using  the  staring  focal  plane  design  case). 

-  The  useful  MTF  might  extend  to  the  first  zero  of  the  detector  MTF  =  1/d. 

-  The  Nyquist  limit  =  1/2d. 

•  A  factor  of  2  difference  exists  in  these  two  naive  estimates  of 
performance,  which  could  correspond  to  a  2X  difference  in  range. 

•  The  performance  usually  lies  between  these  two  limits:  the  “correct” 
performance  estimate  depends  on  the  task  to  be  performed  (or  the  test  to 
be  executed). 

-  In  the  left  figure,  both  the  phone  lines  and  the  truck  grill  show  signs  of  aliasing, 
but  no  problem  of  interpretation  occurs  because  information  is  sufficient. 

-  In  the  right  figure,  aliasing  confuses  image  interpretation. 


lllteH* 
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Digital  Image  of  a  Calibration  Target 


Digital  images  can  be  hard  to  interpret  at 
the  limits  of  resolution. 

Naively,  we  might  think  the  resolution 
task  corresponds  to  one  pixel  on  a  white 
bar,  the  next  pixel  on  the  black  space, 
and  so  on. 

-  But  a  bar  group  of  a  calibration  target  has 
a  wide  frequency  range. 

-  3  rectangular  bars  do  not  make  a  sine 
wave. 

The  left  most  image  of  a  bar  group 
corresponds  to  the  naive  resolution 
case:  detector  width,  W  =  sample 
spacing,  S  =  bar  size 
Remaining  images  are  quite  bizarre 

-  These  represent  cases  where  W  =  S  >  Bar 

-  Equivalently,  the  notional  bar  group 
frequency  (1/(2  Bar))  >  V2  S 

-  We  are  “past  Nyquist,”  in  the  aliasing 
region 


3-bars  (correct) 


Depending  on  the  task 
we  can  still  “resolve” 


The  model  will  distinguish  between  the  detector  width,  W,  and  the  sample  spacing 
S.  For  the  standard  staring  sensor,  S=W.  The  image  depends  on  the  relationship  between 
S  and  W  and  the  width  of  a  bar  in  the  bar  group,  B. 

The  naive  Nyquist  theorem  would  suggest  that  the  target  should  be  resolvable  if  the 
sample  distance  is  less  than  the  bar  width:  S  <  B.  However,  this  leaves  open  the  question 
of  how  good  task  performance  will  be  “beyond  Nyquist”  when  S  >  B. 

The  question  is  complicated  by  the  fact  that  a  bar  target  is  not  a  pure  sine  wave  with 
wavelength  2B;  it  is  a  combination  of  many  frequencies  both  above  and  below  Nyquist. 
A  simple  frequency-based  threshold  will  not  address  the  question. 

As  indicated  in  the  figure,  the  4th  horizontal  bar  group  could  be  interpreted  as  a 
resolved  2-bar  bar  group.  The  task  definition  determines  whether  or  not  this  bar  group 
should  be  considered  to  be  resolved. 
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Task  Definition 

•  For  simple  tasks,  one  can  construct  a  model  of  task  performance. 

•  Consider  defining  resolution  for  bar  targets. 

-  The  result  will  depend  on  whether  2-bar,  3-bar,  4-bar,  etc.,  targets  are  used. 

-  The  system  output  is  examined  to  see  if  it  produces  the  anticipated  up- 
down-up  pattern  of  a  2-bar  target  or  up-down-up-down-up  of  3  bars. 

-  The  task  can  be  varied  by  asking  whether  or  not  the  pattern  has  to  be 
completely  clearly  seen:  should  we  define  task  completion  as  seeing  3-bars 
and  2  spaces  for  a  3-bar  target,  or  seeing  at  least  one  clear  space? 

•  This  may  seem  artificial,  but  it  represents  a  range  of  possible  tasks  and 
can  be  computed  explicitly. 

•  The  following  charts  show  the  probability  of  task  performance  as  a 
function  of  how  much  higher  than  the  Nyquist  limit  one  tries  to  go. 

-  Depends  on  the  MTF  of  the  system  (detector  MTF  and  optics  MTF). 

-  For  some  sample  phases,  the  task  can  be  performed;  for  others,  it  cannot. 

•  In  general,  performance  above  Nyquist  is  stochastic:  for  any  task  and 
frequency  above  Nyquist,  there  is  some  probability  of  performing  the 
task. 

-  With  multiple  opportunities  (as  in  video)  or  with  redundant  information  (the 
truck  grill),  the  Nyquist  limitation  can  be  overcome. 


Allowing  an  N-bar  target  to  be  called  resolved  when  fewer  than  N  bars  can  be 
counted  may  seem  unusual.  However,  it  is  the  basis  of  the  MTDP  figure  of  merit  used  in 
the  German  TRM3  model  in  the  infrared  case  (proposed  as  a  replacement  for  the  MRT). 
It  is  based  on  the  perception  of  the  standard  four-bar  test  pattern  but  accepts  that  fewer 
than  four  bars  are  resolvable  beyond  the  Nyquist  frequency. 
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Results 


•  Basic  tasks  of  resolved  2-bar,  3-bar,  and  4-bar  targets  (seeing  all  bars  and 
spaces). 

•  For  all  tasks,  the  probability  of  task  performance  is  100%  at  and  below 
Nyquist,  but  declines  as  the  frequency  increases. 

-  The  left  chart  shows  results  with  “perfect”  optics  (using  the  detector  MTF  only). 

-  The  right  chart  adds  “matched  optics”  MTF  (with  optics  MTF  zero  located  at 
first  detector  zero). 

•  For  matched  optics  and  the  2-bar  task,  the  average  expected  performance 
will  be  1.25  x  the  Nyquist  limit. 


Results  depend  on  task  definition  and  optics  MTF,  not  just  the  sampling  limit! 


If  the  more  exotic  task  definitions  are  used,  the  performance  extends  even  further. 

For  matched  optics,  and  the  “see  one  space  in  a  four-bar  target”  task,  performance 
extends  to  1.5  x  Nyquist. 

•  50  percent  of  the  possible  sample  phasing  will  permit  accomplishment  of  the 
task. 

•  If  more  than  one  image  is  available,  giving  more  opportunities  to  succeed,  even 
higher  performance  at  higher  spatial  frequencies  is  possible. 

•  This  is  the  source  of  “super-resolution”  (better  to  say  “super-sampling”) 
techniques  but  also  applies  to  video. 
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Reconstruction 


•  The  reconstruction  of  an  image  (how  it  is  displayed  when  there  are  more 
display  pixels  than  image  samples)  has  an  effect  on  its  interpretability. 


-  For  unaliased  images,  there  is  a  unique  best  way  to  reconstruct  an  image 

•  Since  the  sidebands  don’t  overlap,  can  filter  sidebands  completely 

•  Corresponds  to  using  a  sine  function  (sinc(x)=sin(x)/x))  to 
interpolate  between  samples. 

-  For  aliased  images,  there  is  always  some  image  ambiguity. 

•  IDA  document  on  best  least  squares  fit  (IDA  Document  D-4122) 

•  In  practice,  cubic  convolution  interpolation  frequently  is  a 
good  choice. 


•  Reconstruction  typically  becomes  an  issue  when  images  are  examined  at  the 
highest  resolution  -  when  the  image  is  zoomed 


Given  a  sampled  system  to  be  viewed  by  an  observer  or  manipulated  by  an 
algorithm,  the  image  must  be  reconstructed  from  the  samples.  The  samples  are  only  a  list 
of  numbers,  not  an  image.  Reconstruction  can  be  done  poorly  and  must  be  considered 
carefully  to  optimize  the  performance  of  the  sensor.  How  reconstruction  algorithms  are 
designed  depends  on  the  sampling,  detector,  and  optics. 


There  is  some  evidence9  that  the  loss  of  performance  due  to  sampling  artifacts 
depends  primarily  on  aliased  sidebands  at  frequencies  greater  than  Nyquist. 


MTFSampled-system(f')  MTFUnsampled-system(f  '/R')>  R  1  Rout— of —band 


f_ fs/2  I  MF Funsampled- system 

Loo  I MT  Funs  amp  led- system 


SR, 


in-band 


SR 


total 


Loo  I  MT  Funs  amp  led- system  <J-nQH{f)\df 

Lqo  I MT Funs ample d- system  (f-nfs)H(f)\df 


SR out-of-band  ~  SR 


total 


-  SR, 


in-band 


9  Ph.D.  dissertation  of  Steven  K.  Moyer,  Georgia  Institute  of  Technology,  as  well  as  NVESD  and 
Driggers,  Vollmerhausen,  and  0”Kane,  J.  Opt.  Soc.  Am.  A,V ol.  16,  No.  5, May  1999. 
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where  H(f)  is  the  Fourier  transform  of  the  reconstruction  function.  Using  a  reconstruction 
close  to  the  sine  interpolation  would  significantly  reduce  the  effects  since  the  in-band 
spurious  response  would  be  zero.  An  IDA  study10  suggested  that  the  least  squares 
approach  might  provide  an  additional  margin  of  performance  (since  the  reconstruction  is 
superior  to  the  sine  reconstruction),  but  a  complete  human  observers  study  has  not  been 
performed. 


10  Wiener  Interpolation  Function  for  Aliased  Imagery,  IDA  Document  NS-D-4122,  October  2010. 
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Zooming 

•  In  many  practical  cases,  dealing  with  zooming  is  a  matter  of  training. 

-  Don’t  over-zoom:  If  “jaggies”  appear,  one  has  zoomed  too  far. 

-  Jaggies  represent  unphysical  information  from  the  aliased  signal  sidebands. 

•  Why  does  the  size  matter? 

-  One’s  eyes  cannot  see  the  jaggies  unless  one  zooms:  eyes  act  as 
reconstruction  filters. 

•  A  proper  reconstruction  can  reduce  the  need  for  training  and  improve 
performance:  performance  on  calibration  targets  can  be  improved  20%. 


Reconstruction  appears  most  often  in  the  context  of  zooming  in  on  an  image. 
Zooming  implies  that  more  than  one  display  pixel  is  used  for  each  sample,  and  therefore 
a  reconstruction  approach  must  be  made,  no  matter  whether  the  choice  is  apparent  to  the 
user.  It  is  important  to  recall  that  the  appearance  of  the  image  can  depend  on  the  size  of 
the  displayed  image.  The  small  image,  for  which  the  pixel  boundaries  are  difficult  to 
perceive,  is  more  interpretable  than  the  larger  version  of  the  same  image.  In  the  large 
version,  pixilation  (or  jaggies)  is  very  distracting  if  viewed  directly  especially  for  the  45- 
degree  patterns  shown.  (The  square  pixel  reconstruction  is  not  compatible  with  the 
rotated  image). 

Experienced  observers  can  be  trained  not  to  “over  zoom.”  A  general  rule  is  to  stop 
zooming  just  as  jaggies  become  visible.  However,  the  existence  of  jaggies  at  any  zoom 
level  is  an  artifact  of  the  reconstruction  approach.  In  the  terminology  of  in-band  versus 
out-of-band  aliasing,  jaggies,  which  are  seen  when  over-zooming,  are  part  of  the  out-of- 
band  spurious  response. 
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System  Trades:  MTF  versus  Sampling 


•  Reducing  the  overall  MTF  reduces  contrast  but  also  reduces  the  risk 
of  aliasing  artifacts. 


For  a  high  sampling 
rate,  the  high  MTF 
image  is  clearly  the 
superior  image. 


High  MTF/High  Sampling  Rate  Low  MTF/High  Sampling  Rate 


For  a  low  sampling 
rate,  the  low  MTF 
image  may  be  the 
more  useful  image. 

Same  reconstruction 
used  on  all  4  images 


High  MTF/Low  Sampling  Rate 


Low  MTF/Low  Sampling  Rate 


The  effect  is  illustrated  with  the  task  of  reading  license  plates.  The  high  MTF  when 
combined  with  a  high  sampling  rate  is  the  best  image.  However,  if  the  sampling  rate  is 
lowered,  the  high  MTF  image  has  the  higher  contrast  but  the  low  MTF  image  is  more 
readable. 
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System  trades:  Reconstruction  vs.  MTF 


•  These  sensor  trades  depend  on  how  the  digital  information  is  displayed. 
High  MTF/Low  Sampling  Rate  Low  MTF/Low  Sampling  Rate 


This  section  addresses  the  question  of  how  digital  samples  should  be  represented  in 
a  final  image.  This  is  directly  relevant  for  humans  observing  the  image,  but  this  can  also 
matter  if  multiple  images  are  to  be  combined.  The  images  on  the  top  row  are  low  sample 
rate  images  with  high  and  low  MTF.  In  this  case,  the  sample  values  are  displayed 
(reconstructed)  with  more  than  one  display  pixel  (pixel  is  a  word  for  a  “picture  element” 
and  properly  should  be  used  only  in  the  context  of  the  display  of  an  image)  for  each 
sample.  Each  sample  is  represented  by  a  small  square  of  pixels,  each  having  the  same 
pixel  value.  This  results  in  the  appearance  of  “pixilation.”  The  severe  aliasing  of  the 
upper  left  image  may  lead  to  the  lower  MTF  upper  right  image  being  preferred. 

The  second  row  uses  the  same  sampled  data  and  again  represents  each  sample  with 
more  than  one  pixel.  However,  in  this  case,  the  pixel  values  are  not  the  same  for  each 
pixel  in  the  small  square:  they  vary  according  to  a  reconstruction  algorithm.  The 
distracting  pixelation  effects  are  removed  and  the  images  may  be  more  interpretable. 

It  is  important  to  stress  that  the  lower  row  is  not  “blurred”  compared  to  the  upper 
row.  The  appearance  of  sharp  edges  (particularly  in  the  upper  right),  which  may  suggest 
higher  sharpness,  is  an  illusion.  The  sharp  edges  are  an  artifact  of  the  “pixel  replication” 
algorithm  used  to  display  the  image  and  are  not  a  real  part  of  the  image. 

As  the  images  show,  choosing  a  reconstruction  algorithm  can  be  an  important  factor 
in  image  interpretability. 
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Consequences 

•  A  complete  MTF  characterization  and  not  merely  Nyquist  arguments  are 
needed  to  evaluate  the  anticipated  performance. 

•  Performance  at  frequencies  below  Nyquist  are  less  likely  to  be  disrupted 
by  sampling  effects,  but  can  be  (Moire  patterns  are  low-frequency 
artifacts  produced  by  strong  high-frequency  content  of  the  scene). 

•  Performance  above  Nyquist  will  define  the  ultimate  limiting  performance 
(maximum  range  at  which  any  specified  task  can  be  performed). 

•  Performance  may  range  from  1.25  to  1.5  x  the  Nyquist  limit. 

-  Multiple  images  (giving  multiple  looks)  will  extend  performance. 

-  Even  single  images  with  multiple  effective  sample  phases  may 
permit  task  performance. 

-  Improvements  may  extend  to  2  *  Nyquist. 

•  Performance  above  Nyquist  will  depend  crucially  on  the  task. 

-  Toy  tasks  such  as  reading  calibration  targets  are  susceptible  to  analysis. 

-  The  relationship  of  these  tasks  to  military  tasks  is  uncertain. 

-  Testing  should  endeavor  to  have  a  mix  of  analyzable  tasks  (for  system 
characterization)  and  operational  tasks. 

-  Operational  tasks  at  “higher”  levels  such  as  target  identification  (rather  than 
simply  detection)  are  hard  to  define  in  fully  analytical  terms. 


Sampling,  Aliasing,  and  Reconstruction  Summary 

•  Aliasing  is  an  almost  inescapable  feature  of  staring  focal  plane  array  sensors  and 
refers  to  the  artifacts  associated  with  sampling  systems.  They  can  range  from 
Moire  patterns  to  ambiguously  interpreted  images  of  objects  with  complicated 
patterns  of  light  and  dark. 

•  The  presence  of  aliasing  in  a  system  gives  rise  to  a  commonly  used  rule  of 
thumb  based  on  the  Nyquist  frequency,  identifying  the  resolution  of  a  system 
with  the  size  of  a  sample  (or  pixel)  on  the  target.  This  underrates  the  importance 
of  other  determiners  of  resolution  such  as  the  modulation  transfer  function  or 
MTF.  Performance  may  be  poorer  than  that  suggested  by  Nyquist  or  better  than 
Nyquist,  depending  on  the  task.  The  combination  of  MTF  and  aliasing  can  lead 
to  some  unintuitive  tradeoffs:  poorer  MTF  may  actually  improve  performance  in 
some  tasks  if  aliasing  is  a  significant  problem. 

•  For  imaging  systems  with  aliasing,  the  display  of  the  sensor  information 
becomes  important,  especially  if  the  image  is  zoomed  so  that  one  display  pixel  is 
used  to  represent  a  single  sensor  sample  value.  An  improper  reconstruction  of 
the  sensor  data  can  reduce  performance.  For  all  aliased  systems,  an 
understanding  of  the  reconstruction  algorithm  is  an  important  aspect  of  any 
system  including  a  human  operator. 
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G.  Color  Aliasing  in  Bayer  Cameras 


Seeing  Color  in  Black  and  White 


U.S.  Calibration  Target 


Color  aliasing  is  a  common  property  of  commercial  Bayer  reconnaissance 
cameras  (but  not  amateur  cameras  like  Nikon,  Canon,  Olympus,  and  Sony) 


The  purpose  of  this  section  is  to  discuss  additional  issues  that  arise  from  the  use  of 
Bayer  pattern  cameras.  In  addition  to  the  aliasing  usually  associated  with  digital  sensors, 
Bayer  pattern  cameras  may  have  color  aliasing. 

Just  as  the  aliasing  of  a  monochromic  camera  may  be  addressed  by  different 
reconstruction  functions,  the  effects  of  the  color  aliasing  depend  on  the  deBayering  (or 
demosaicing)  algorithm  used  to  generate  the  interpolated  red,  green,  and  blue  values. 

In  the  image  shown,  the  combination  of  sensor  MTF  and  demosaicing  algorithm 
have  produced  severe  color  aliasing.  The  white  bars  on  the  black  background  of  the 
calibration  target  appear  to  have  colors,  a  blue-green  and  orange.  At  the  triangular 
fiducial  marks,  the  bars  of  the  target  are  blurred  together  to  form  blue-green  or  orange 
blurs. 
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Color  Systems 


*  Full  color  systems  have  multiple  lenses 
(Intergraph  DMC  ,  Vexel  UltraCam)  or  beam 
splitters  (Leica  ADS-80)  with  separate  arrays 
for  each  color. 

*  Bayer  systems  use  arrays  of  mixed  red, 
green,  and  blue  detectors  (“pixels”). 

-  Cheaper 

-  Exploits  consumer  electronics  for  arrays 

-  Large  fields  of  view  using  multiple  Bayer  arrays 

-  Lower  storage  and  bandwidth  requirements. 

*  Bayer  systems  have  color  aliasing  in  addition 
to  ordinary  aliasing:  the  space  between  red 
samples  is  2  x  detector  size. 

-  For  a  “4  megapixel”  array,  only  1  million  red,  1 
million  blue,  and  2  million  green. 

-  An  algorithm  is  required  to  produce  4  million 
R,G,  B  values. 

-  The  algorithm  will  affect  system  performance, 
including  resolution. 


Full  color  systems  have  separate  focal  planes  for  each  color  (red,  green,  blue  and,  in 
some  cases,  near  infrared).  Since  the  focal  planes  are  separate,  they  require  separate 
optical  paths. 

In  some  cases,  this  is  performed  by  completely  separate  lens  systems  as  illustrated 
in  the  upper  photograph.  On  the  left,  the  UltraCam  by  Vexel  uses  separate  optics  for  red, 
green,  blue,  and  infrared,  supplemented  by  four  additional  panchromatic  lenses  for  higher 
resolution.  On  the  right  is  the  DMC  II  by  Intergraph,  which  uses  one  panchromatic 
camera  and  the  four  additional  red,  green,  blue,  and  near  infrared. 

For  line-scanners,  such  as  the  Leica  ADS-40,  a  single  primary  optical  system  can 
include  a  beam  splitter  separating  the  light  into  different  bands  directed  to  different  rows 
of  detectors. 

Bayer  pattern  cameras  can  be  less  complex  and  expensive  than  full-color  cameras. 
By  providing  color  information  in  a  single  focal  plane  (at  the  cost  of  increasing  the 
sample  distance  for  each  color),  significant  savings  can  be  made. 

For  both  types  of  color  cameras,  additional  complexity  may  be  introduced  to 
provide  a  wider  field  of  view.  The  lower  photograph  shows  a  multi-lens  system 
[manufactured  by  the  Russian  firm  NPO  KSI  (Scientific  Production  Union  -  KSI)]  that 
uses  several  Bayer  pattern  focal  planes  with  each  lens,  interleaving  them  to  form  a  final 
image. 
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Bayer  Patterns 


Providing  a  full  set  of  R,  G,  and  B  values  at  each  site  is  called 
“deBayering”  or  “demosaicing.” 

There  are  many  different  approaches  to  deBayering 

-  Linear  interpolations: 

•  Band-by-band  (missing  G  pixels  depend  only  on  G  pixels) 

•  Cross-band  (missing  G  pixels  depend  on  G  and  R/B  pixels) 

-  Hue  based  -  Assuming  G  pixels  provide  brightness  and  then  use 
R/B  pixels  to  determine  hue  (colorizing) 

-  Gradient  estimators  -  Avoiding  averaging  across  an  edge 

-  Other  approaches  use  non-linear  techniques  (median  filters) 

-  Etc. 


■■■■ 

mnnn 


■ 

new 

new 

new 

•  Approaches  can  be  simple,  using  only  the  nearest  neighbors 
or  complex  with  a  larger  number  of  measured  values 
involved  I+1/2I 


B  at  green  in 
B  row,  R  column 


DeBayering  is  the  process  of  generating  by  some  algorithm  estimates  of  the  sample 
values  that  would  have  been  produced  by  a  full-color  array.  For  example,  for  each  red 
sample  value,  three  additional  sample  values  must  be  produced. 

An  enormous  range  of  algorithms  exist  for  deBayering.  However,  in  contrast  to  the 
Wiener  filter,  there  is  no  established  theoretical  basis  for  choosing  one  algorithm  over 
another.  The  filters  can  be  linear:  averaging  sample  values  from  the  measured  detectors 
to  interpolate  new  values.  This  can  be  done  for  each  color  separately,  but  it  has  been 
found  useful  by  some  authors  to  mix  colors,  using  the  intensity  of  green  samples  as 
indicators  of  brightness  and  using  those  values  to  influence  nearby  red  interpolated 
values.  An  example  will  be  shown  in  a  subsequent  chart. 

Another  algorithm  uses  the  more  numerous  green  values  to  estimate  the 
panchromatic  brightness  of  the  scene,  and  then  using  the  red  and  blue  values  to  estimate 
the  color.  This  may  work  on  scenes  that  are  not  too  dramatically  colored. 

Other  algorithms  try  to  identify  gradients  or  edges  to  avoid  interpolating  across  a 
gap  in  brightness. 
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Demosaicing  Causes  False  Colors 


•  The  degree  of  false  colors  depends  on  the 
complexity  of  the  algorithm. 

•  The  better  algorithms  require  significantly 
more  processing. 

•  For  all  algorithms  there  is  an  inevitable  loss  of 
resolution  depending  on  the  image. 


The  source  of  the  blue-green  and  orange  errors  on  the  black-and-white  calibration 
target  can  be  understood  in  the  simple  nearest-neighbor  linear  deBayering  algorithm. 
Imagine  that  the  detector  locations  fell  on  the  white  stripes  as  shown  at  the  left,  with  only 
blue  and  green  detectors  on  the  white  stripes  and  red  detectors  always  on  the  black 
background. 

Consider  a  red  site  between  two  of  the  white  stripes.  There  is  no  red  value  at  that 
site,  so  in  the  simple  algorithm,  the  red  value  in  the  final  result  will  be  zero.  However, 
both  blue  and  green  values  will  be  assigned  since  there  are  blue  and  green  neighbors. 

Similarly,  along  the  white  stripe,  no  red  values  will  be  assigned,  but  both  blue  and 
green  values  will  be  assigned  at  each  point. 

The  result  will  be  a  blue-green  color  assigned  along  the  white  bars  and  along  the 
black  spaces  between  the  bars.  The  bars  will  disappear  into  a  blue-green  smear. 
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Interpolation  Errors 


Original  and  two  Bayered  then  deBayered  images 


Original  (full-color  camera) 


Figures  reproduced  from  Malvar,  et  al. 


Bayered:  DeBayered  with 
simple  linear  interpolation 


Bayered:  DeBayered  with 
Microsoft  interpolation 


The  left  figure  shows  a  high-resolution  full-color  image.  A  Bayer  camera  version 
can  be  produced  by  selecting  only  the  red,  green,  or  blue  values  from  each  pixel. 

The  middle  image  uses  simple  nearest  neighbor  linear-interpolation.  The  image 
shows  characteristic  blue-green  and  orange  structure  along  the  bars  of  the  shutters.  The 
authors’  own,  more  elaborate  algorithm  eliminates  or  diminishes  the  color  aliasing  errors 
and  is  shown  on  the  right. 

Just  as  some  image  interpreters  prefer  the  j  aggies  of  pixel-replication  zooming 
(primarily  because  of  familiarity  but  also  because  allows  recognition  of  over-zooming), 
some  interpreters  prefer  the  image  with  color  aliasing  to  the  apparently  superior  image 
produced  by  the  Microsoft  algorithm. 

As  in  the  case  of  zooming,  there  is  no  way  to  avoid  some  algorithm  for  deBayering, 
and  there  is  no  reason  not  to  use  more  sophisticated  algorithms. 
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Exotic  Bayer  Artifacts 


•  Colors  can  go  beyond  blue-green  and  orange. 

-  Notice  that  the  horizontal  bars  on  the  left  are 
apparently  vertical  (and  orange) 

•  The  Bayer  artifacts  may  extend  over  long 
distances  depending  on  how  large  a  region  is 
used  for  the  interpolation. 

-  In  the  lower  image,  the  top  row  of  bars  has  a 
small  separation  between  groups  of  bars;  the 
demosaicing  algorithm  connects  across 
different  bar  groups  producing  the  dramatic 
colors. 

-  By  separating  the  bar  groups,  the  Bayer 
anomalies  are  removed. 

•  The  Bayer  artifacts  will  depend  on  the 
algorithm  and  the  objects  imaged! 


The  presence  of  color  artifacts  depends  upon  both  the  demosaicing  algorithm  and 
the  scene.11  In  general,  algorithms  with  less  color  aliasing  are  more  expensive 
computationally. 

Color  Aliasing  in  Bayer  Cameras  Summary 

•  The  use  of  Bayer  pattern  color  camera  in  imaging  systems  in  order  to  reduce 
cost  and  data  rate  requirements  may  introduce  additional  dependence  on 
algorithmic  choices.  In  contrast  to  full-color  cameras,  a  Bayer  camera  does  not 
have  a  color  detector  for  all  three  colors  (red,  green,  blue)  at  each  location  (a  12 
“megapixel”  Bayer  sensor  has  6  million  green,  3  million  red,  and  3  million  blue 
detectors).  An  algorithm  is  used  to  interpolate  the  missing  values  to  produce  a 
complete  three-color  image. 

This  color  aliasing  is  more  severe  than  the  aliasing  intrinsic  to  any  staring  focal 
plane.  Fortunately,  in  the  natural  world,  the  color  information  is  usually 
relatively  immune  to  aliasing  effects.  However,  a  large  number  of  different 
algorithms  of  varying  complexity  may  be  employed  by  a  sensor  manufacturer. 
Depending  on  the  tasks  to  be  performed  and  the  details  of  the  image,  the 
algorithm  may  induce  new  artifacts  that  complicate  the  evaluation  of  the  system. 

Ideally,  the  user  could  be  given  some  control  over  the  interpolation  algorithm  to  match  it 
to  the  task. 


11  Patrick  C.  Hytla,  Demosaicing  Algorithm  Performance  for  the  Open  Skies  Treaty  (White  Paper),  Sensor 
Systems  Division,  University  of  Dayton  Research  Institute,  Dayton,  OH,  May  25,  201 1 . 
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Appendix  A 
Acronyms  and  Terms 


Acronyms 

3D 

three  dimensional 

ADC 

Analog  to  Digital  Conversion 

A1203 

sapphire 

AlGaAs 

aluminum  gallium  arsenide 

C02 

carbon  dioxide 

CTF 

Contrast  Threshold  Function 

DSP 

Defense  Support  Program 

EO 

electro-optical 

FAR 

false  alarm  rate 

FFT 

fast  Fourier  transform 

FLIRS 

forward-looking  infrared  systems 

FOV 

field  of  view 

FPA 

focal  plane  array 

GaAs 

gallium  arsenide 

Ge 

germanium 

GIQE 

General  Image  Quality  Equation 

GSD 

ground  sample  distance 

H20 

water 

HgCdTe 

mercury  cadmium  telluride 

ICBM 

intercontinental  ballistic  missile 

IFOV 

instantaneous  field  of  view 

IMM 

interacting  multiple  model 

InGaAs 

indium  gallium  arsenide 

InSb 

indium  antimonide 

IR 

infrared 

IRFPA 

infrared  focal  plane  array 

IRST 

infrared  search  and  track 

ISR 

intelligence,  surveillance,  and  reconnaissance 

A-l 


L 

LOS 

LWIR 

MHT 

MODTRAN 

MRT 

MTDP 

MTF 

MTI 

MWIR 

NEI 

NETD 

NIIRS 

NIR 

NUC 

NVESD 

NV-IPM 

NVThermlP 

OBC 

OH 

Pd 

RER 

SAR 

SBIRS 

SGR 

SNR 

SSCamlP 

SWaP 

SWIR 

HP 

UAV 

UDRI 

UV 

VLWIR 


jam 


emitted  radiance 
line  of  sight 

long-wavelength  infrared 

multiple  hypothesis  tracker 

MODerate  resolution  atmospheric  TRANsmission 

minimum  resolvable  temperature 

minimum  temperature  difference  perceived 

Modulation  Transfer  Function 

moving  target  indicator 

mid-wavelength  infrared 

noise  equivalent  irradiance 
noise  equivalent  temperature  difference 
National  Imagery  Interpretability  Rating  Scale 
near-infrared 

Non-uniformity  Correction 

Night  Vision  and  Electronic  Sensors  Directorate 

Night  Vision  Integrated  Performance  Model 

Night  Vision  Thermal  and  Image  Processing 

Optical  Bar  Camera 
hydroxyl 

probability  of  detection 

relative  edge  response 

synthetic  aperture  radar 
Space-Based  Infrared  System 
sky-to-ground  ratio 
signal-to-noise  ratio 

Solid  State  Camera  and  Image  Processing 
Size,  Weight,  and  Power 
short- wavelength  infrared 

Targeting  Task  Performance 

unmanned  aerial  vehicle 

University  of  Dayton  Research  Institute 

ultraviolet 

very  long-wavelength  infrared 
micrometer 


A-2 


Terms 

Aliasing 


Bayer  pattern  color  camera 

deBayerization  (demosaicing) 

Detector 
Detector  size 

Detector  pitch 

Full  color  camera 

Luminance 


Minimum  Resolvable 
Temperature  (MRT) 


Modulation  Transfer 
Function  (MTF): 

Nyquist  frequency 


Pixel 

Radiance 

Reconstruction 

Sample 
Sampling  rate 


A  consequence  of  modem  digital  sampling.  Introduces 
distortions  into  the  image. 

Focal  plane  includes  a  mixture  of  detectors  for  red, 
green,  and  blue. 

Interpolating  missing  red,  green,  and  blue  values  for  a 
Bayer  camera. 

Photo-sensitive  device  for  measuring  incident  radiation. 
Detectors  are  typically  square  with  size  ranging  from  4 
(visible)  to  25  m  (IR) 

The  spacing  between  detectors.  The  detector  size  is  < 
the  detector  pitch. 

Separate  focal  planes  for  red,  green,  and  blue  detectors. 
Contrast  with  Bayer  camera. 

A  measure  of  emitted  or  reflected  light  per  area, 
expressed  in  units  of  foot- Lamberts  or  candela  per 
square  meter. 

An  IR  performance  measure  giving  the  temperature 
difference  required  to  resolve  features  at  different 
spatial  frequency. 

The  frequency  response  of  an  optical  system  including 
the  optics,  lenses,  vibration,  etc. 

'/2  the  sampling  rate  of  a  digital  system.  If  the  highest 
spatial  frequency  in  a  scene  is  less  than  the  Nyquist 
frequency,  the  system  is  unaliased. 

Properly,  a  displayed  “picture  element.”  Loosely,  a 
detector  or  sample. 

The  energy  radiated  or  reflected  by  an  object. 

The  process  of  taking  digital  data  and  displaying  it  or 
interpolating  it. 

The  result  of  digitizing  the  output  of  a  detector. 

The  spacing  between  digital  samples.  May  be 
expressed  in  angular  terms  or  spatial  terms. 
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